Optimizing checkpoint locations along an insertion trajectory of a medical instrument using data analysis

ABSTRACT

Provided are computer-implemented methods and systems for generating and/or utilizing model(s) for determining optimized checkpoint locations along a trajectory in an image-guided procedure for inserting a medical instrument to a target in a body of a patient based, inter alia, on data related to an automated medical device and/or to operation thereof.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Bypass Continuation of PCT Patent Application No.PCT/IL2021/050441 having International filing date of Apr. 19, 2021,which claims the benefit of priority of U.S. Provisional PatentApplication No. 63/012,196, filed Apr. 19, 2020, the contents of whichare all incorporated herein by reference in their entirety

FIELD OF THE INVENTION

The present invention relates to computer-implemented methods andsystems for collecting data related to operation of automated medicaldevices, and utilization of the data to generate algorithms to determineand/or recommend optimized checkpoint locations along a trajectory forinserting a medical instrument toward an internal target.

BACKGROUND

Various diagnostic and therapeutic procedures used in clinical practiceinvolve the insertion of medical instruments, such as needles andcatheters, percutaneously to a subject's body, and in many cases furtherinvolve the steering of the medical instruments within the body, toreach a target region. The target region can be, for example, a lesion,a tumor, an organ and/or a vessel. Examples of procedures requiringinsertion and steering of such medical instruments include vaccinations,blood/fluid sampling, regional anesthesia, tissue biopsy, catheterinsertion, cryogenic ablation, electrolytic ablation, brachytherapy,neurosurgery, deep brain stimulation, various minimally invasivesurgeries, and the like.

The guidance and steering of medical instruments in the body is acomplicated task that requires good three-dimensional coordination,knowledge of the patient's anatomy and a high level of experience. Thus,image-guided automated (e.g., robotic) systems have been proposed forperforming these functions.

Some automated systems are based on manipulating robotic arm(s) and someutilize a robotic device which can be attached to the patient's body orpositioned in close proximity thereto. These automated systems typicallyassist the physician in aligning the medical instrument with a selectedinsertion point at a desired insertion point and the insertion itself iscarried out manually by the physician. Some automated systems furtherinclude an insertion mechanism that can insert the instrument toward thetarget, typically in a linear manner. More advanced automated systemsfurther include non-linear steering capabilities, as described, forexample, in U.S. Pat. Nos. 8,348,861, 8,663,130 and 10,507,067, and inco-owned U.S. Pat. No. 10,245,110, co-owned U.S. Patent ApplicationPublication No. 2019/290,372, and co-owned International PatentApplication No. PCT/IL2020/051219, all of which are incorporated hereinby reference in their entireties.

During the operation of such automated medical devices in variousprocedures and in various settings, a large amount of related data isaccumulated. The utilization of such data to improve and enhance theoperation and clinical value of these automated devices, as well as topredict and/or detect clinical conditions, and specifically, clinicalcomplications, may ultimately improve the health and safety of thepatients.

Thus, there is a need in the art for methods and systems for collectingand processing the data related to and/or generated by automated medicaldevices, and for generating and implementing data-analysis algorithms(e.g., artificial intelligence (AI) models) that can utilize theaccumulated data to provide operating recommendations, operatinginstructions, functionality enhancements, clinical evaluations andpredictions, etc.

SUMMARY

According to some embodiments, the present disclosure is directed tosystems and computer-implemented methods for the collection of varioustypes of datasets related to and/or obtained from operation of automatedmedical devices and the consequent manipulation and/or utilization ofthe data, to generate algorithms (or—models) to one or more of: affect,control and/or manipulate the operation of automated devices, generaterecommendation to users of automated devices, and/or predict clinicalconditions and/or complications, based on at least some of the collecteddata and/or parameters derived therefrom. In some embodiments, thecomputerized methods may utilize specific algorithms which may begenerated using machine learning tools, deep learning tools, datawrangling tools, and, more generally, AI and data analysis tools. Insome embodiments, the specific algorithms may be implemented usingartificial neural network(s) (ANN), such as convolutional neural network(CNN), recurrent neural network (RNN), long-short term memory (LSTM),auto-encoder (AE), generative adversarial network (GAN),Reinforcement-Learning (RL) and the like, as further detailed below. Inother embodiments, the specific algorithms may be implemented usingmachine learning methods, such as support vector machine (SVM), decisiontree (DT), random forest (RF), and the like. Both “supervised” and“unsupervised” methods may be implemented.

In some embodiments, data is collected during or resulting fromprocedures performed by the automated medical devices. In someembodiments, the collected data may be used, to generate analgorithm/model, which may consequently provide, for example,instructions, enhancements or recommendations regarding variousoperating parameters and/or other parameters related to automatedmedical devices. Thus, based at least on some of the collected primarydata (also referred to as “raw data”) and/or metadata and/or data and/orfeatures derived therefrom (“manipulated data”) and/or annotationsgenerated manually or automatically, a data-analysis algorithm may begenerated, to provide output that can enhance the operation of theautomated medical devices and/or the decisions of the users (e.g.,physicians) of such devices.

In some exemplary embodiments, the automated medical devices are devicesfor insertion and steering of medical instruments (for example, needles,introducers or probes) in a subject's body for various diagnostic and/ortherapeutic purposes. In some embodiments, the automated insertiondevice may utilize real-time instrument position prediction andreal-time trajectory updating, as disclosed, for example, inabovementioned co-owned International Patent Application No.PCT/IL2020/051219. For example, when utilizing real-time trajectoryupdating and steering, the most effective spatio-temporal and safe routeof the medical instrument to the target within the body may be achieved.Further, safety may be increased as it reduces the risk of harmingnon-target regions and tissues within the subject's body, as thetrajectory update may take into account obstacles or any other regionsalong the route, and moreover, may take into account changes in thereal-time location of such obstacles. Additionally, such automaticsteering may improve the accuracy of the procedures, thus enablingreaching small and hard to reach targets. This can be of particularimportance in early detection of malignant neoplasms, for example. Inaddition, it provides increased safety for the patient, as there is asignificant lower risk of human error. Further, such a procedure may besafer for the medical personnel, as it may minimize their exposure toradiation and/or pathogens during the procedure. In some embodiments,the automated medical devices are configured to insert andsteer/navigate a medical instrument (in particular, the tip of themedical instrument) in the body of the subject, to reach a target regionwithin the subject's body, to perform various medical procedures. Insome embodiments, the operation of the medical devices may be controlledby at least one processor configured to provide instructions, inreal-time, to steer the medical instrument and the tip thereof, towardthe target, according to a planned and/or the updated trajectory. Insome embodiments, the steering may be controlled by the processor, via asuitable controller. In some embodiments the steering may be controlledin a closed-loop manner, whereby the processor generates motion commandsto the steering device via a suitable controller and receives feedbackregarding the real-time location of the medical instrument and/or thetarget. In some embodiments, the processor(s) may be able to predict thelocation and/or movement pattern of the target. AI-based algorithm(s)may be used to predict the location and/or movement pattern of thetarget. In some embodiments, the automated medical device may beconfigured to operate in conjunction with an imaging system. In someembodiments, the imaging system may include any type of imaging system,including, but not limited to: X-ray fluoroscopy, CT, cone beam CT, CTfluoroscopy, MM, ultrasound, or any other suitable imaging modality. Insome embodiments, the processor is configured to calculate a trajectoryfor the medical instrument based on a target, entry point and,optionally, obstacles en route (such as bones or blood vessels), whichmay be manually marked by the user, or automatically identified by theprocessor, on one or more obtained images.

In some embodiments, the primary datasets collected and utilized by thesystems and methods disclosed herein may include several types of setsof primary data, including, for example, clinical related dataset,patient related dataset, device related dataset and/or administrativedataset. The collected datasets may then be manipulated/processed,utilizing data analysis algorithms, machine learning algorithms and/ordeep learning algorithms, to generate an algorithm or a model, which mayoutput, inter alia, recommendations and/or operating instructions forthe automated medical device, to thereby enhance their operation.

According to some embodiments, the collected datasets and/or the dataderived therefrom may be used for the generation of a training set,which may be part of the generated algorithm/model, or utilized for thegeneration of the model/algorithm and/or the validation or updatethereof. In some embodiments, the training step may be performed in an“offline” manner, i.e., the model may be trained/generated based on astatic dataset. In some embodiments, the training step may be performedutilizing an “online” or incremental/continuous manner, in which themodel is continuously updated with every new incoming data.

According to some embodiments, there is thus provided acomputer-implemented method of generating a checkpoint locations modelfor optimizing locations of checkpoints along a trajectory in animage-guided procedure for inserting a medical instrument toward atarget in a body of a patient, the method including:

collecting one or more datasets, at least one of the one or moredatasets being related to an automated medical device configured tosteer a medical instrument toward a target in the body of a patientand/or to operation thereof;

creating a training set which may include at least a portion of the oneor more datasets and one or more target parameters relating tocheckpoint locations along a trajectory in one or more previousimage-guided procedures for inserting a medical instrument toward atarget in a body of a patient;

training the checkpoint locations model to predict checkpoint locationsusing the training set;

calculating a checkpoint locations prediction error; and

optimizing the checkpoint locations model using the calculatedcheckpoint locations prediction error.

According to some embodiments, the one or more datasets may furtherinclude one or more of: clinical procedure related dataset, patientrelated dataset and administrative related dataset.

According to some embodiments, the automated medical device relateddataset may include parameters selected from: entry point, insertionangles, target position, target position updates, target contour and/orbounding box and/or location, “no-fly” zones masks and/or bounding boxesand/or locations, organs segmentation masks and/or bounding boxes and/orlocations, tissues segmentation mask and/or bounding boxes and/orlocations, blood vessels mask and/or bounding boxes and/or locations,planned trajectory, trajectory updates, real-time positions of themedical instrument, number of checkpoints along the planned and/orupdated trajectory, checkpoint locations, checkpoint locations updates,checkpoint errors, position of the automated medical device, steeringsteps timing, procedure duration, steering phase duration, procedureaccuracy, target error, medical images, medical imaging parameters perscan, radiation dose per scan, total radiation dose in steering phase,total radiation dose procedure, sensor(s) measurements, errors indicatedduring the steering procedure, software logs, motion control traces,automated medical device registration logs, medical instrument detectionlogs, homing and BIT results, or any combination thereof.

According to some embodiments, at least one of the parameters of theautomated medical device related dataset may be used as a trainingparameter (data annotation).

According to some embodiments, the clinical procedure related datasetmay include parameters selected from: medical procedure type, targetorgan, target size, target type, type of medical instrument, dimensionsof the medical instrument, clinical complications before, during and/orafter the procedure, complications detection time, adverse eventsbefore, during and/or after the procedure, respiration signals of thepatient, or any combination thereof.

According to some embodiments, the medical procedure type may beselected from: fluid sampling, regional anesthesia, tissue biopsy,catheter insertion, cryogenic ablation, electrolytic ablation,brachytherapy, neurosurgery, deep brain stimulation, minimally invasivesurgery, or any combination thereof.

According to some embodiments, the patient related dataset may includeparameters selected from: age, gender, race, medical condition, medicalhistory, vital signs before, after and/or during the procedure, bodydimensions, pregnancy, smoking habits, demographic data, or anycombination thereof.

According to some embodiments, the administrative related dataset mayinclude parameters selected from: institution, physician, staff, systemserial number, disposable components used in the procedure, softwareversion, operating system version, configuration parameters, or anycombination thereof.

According to some embodiments, one or more of the parameters of the oneor more datasets may be configured to be collected automatically.

According to some embodiments, the checkpoint locations model may begenerated utilizing artificial intelligence tools. In some embodiments,the artificial intelligence tools may include one or more of: machinelearning tools, data wrangling tools, deep learning tools, artificialneural network (ANN), deep neural network (DNN), convolutional neuralnetwork (CNN), recurrent neural network (RNN), long short term memorynetwork (LSTM), decision trees or graphs, association rule learning,support vector machines, inductive logic programming, Bayesian networks,instance-based learning, manifold learning, sub-space learning,dictionary learning, reinforcement learning (RL), generative adversarialnetwork (GAN), clustering algorithms, or any combination thereof.

According to some embodiments, training the checkpoint locations modelmay include using one or more of: loss function, Ensemble Learningmethods, Multi-Task Learning, Multi-Output regression and Multi-Outputclassification.

According to some embodiments, the method of generating a checkpointlocations model for optimizing locations of checkpoints may furtherinclude:

executing one or more individual models using at least a portion of theone or more datasets and a checkpoint locations prediction generated bythe checkpoint locations model; and

obtaining one or more predictions from the one or more individualmodels.

According to some embodiments, the method of generating a checkpointlocations model may further include:

calculating a loss function using a checkpoint locations predictionerror and the one or more predictions generated by the one or moreindividual models; and

optimizing the checkpoint locations model using the loss function.

According to some embodiments, the method of generating a checkpointlocations model may further include training the one or more individualmodels.

According to some embodiments, the one or more individual models mayinclude a model for predicting an accuracy of an image-guided insertionprocedure.

According to some embodiments, the one or more individual models mayinclude a model for predicting a radiation dose emitted during animage-guided insertion procedure, or part thereof.

According to some embodiments, the one or more individual models mayinclude a model for predicting a duration of an image-guided insertionprocedure, or part thereof.

According to some embodiments, the one or more individual models mayinclude a model for predicting a risk of an image-guided insertionprocedure.

According to some embodiments, calculating the loss function may includeminimizing the checkpoint locations prediction error.

According to some embodiments, calculating the loss function may includeminimizing the predicted radiation dose.

According to some embodiments, calculating the loss function may includeminimizing the predicted duration.

According to some embodiments, calculating the loss function may includeminimizing the predicted risk.

According to some embodiments, calculating the loss function may includemaximizing the predicted accuracy of the image-guided insertionprocedure.

According to some embodiments, the method of generating a checkpointlocations model may further include adjusting one or more coefficientsof one or more terms used in the calculation of the loss function, theone or more terms being associated with at least one of the checkpointlocations prediction error and the one or more predictions generated bythe one or more individual models.

According to some embodiments, the adjusting of the one or morecoefficients may be executed during training of the checkpoint locationsmodel.

According to some embodiments, the adjusting of the one or morecoefficients may be executed during execution of the checkpointlocations model.

According to some embodiments, generating the checkpoint locations modelmay be executed by a training module which may include a memory and oneor more processors.

According to some embodiments, the automated medical device may beconfigured to steer the medical instrument toward the target such thatthe medical instrument traverses a non-linear trajectory within the bodyof the patient.

According to some embodiments, the automated medical device may beconfigured to allow real-time updating of a trajectory of the medicalinstrument.

According to some embodiments, there is provided a system for generatinga checkpoint locations model for optimizing locations of checkpointsalong a trajectory in an image-guided procedure for inserting a medicalinstrument to an internal target, the system includes:

a training module including:

-   -   a memory configured to store the one or more datasets; and    -   one or more processors configured to execute the method of any        one of the method of generating a checkpoint locations model        disclosed herein.

According to some embodiments, the training module may be located on aremote server, an “on premise” server or a computer associated with theautomated medical device.

According to some embodiments, the remote server may be a cloud server.

According to some embodiments, there is provided a computer-implementedmethod of utilizing a checkpoint locations model for optimizinglocations of checkpoints along a trajectory in an image-guided procedurefor inserting a medical instrument to a target in a body of a patient,the method includes:

collecting one or more new datasets, at least one of the one or more newdatasets being related to an automated medical device configured tosteer a medical instrument toward a target in a body of a patient and/orto operation thereof and including one or more images of a region ofinterest and a planned trajectory for the medical instrument from anentry point to the target;

detecting one or more tissue boundaries in the one or more images;

executing the checkpoint locations model using at least a portion of theone or more new datasets;

obtaining an output of the checkpoint locations model; and

setting one or more checkpoints along the planned trajectory based onthe output of the checkpoint locations model.

According to some embodiments, the method of utilizing a checkpointlocations model may further include obtaining from a user at least oneof a confirmation to the set one or more checkpoints and an adjustmentthereto.

According to some embodiments, the method of utilizing a checkpointlocations model may further include defining one or more sections alongthe planned trajectory in which no checkpoints are to be positioned, soas to allow the medical instrument to be continuously advanced along theone or more sections.

According to some embodiments, the method of utilizing a checkpointlocations model may further include estimating a scan volume and aradiation dose per checkpoint.

According to some embodiments, the automated medical device may beconfigured to allow real-time updating of a trajectory of the medicalinstrument.

According to some embodiments, if at least one of a position of thetarget and the planned trajectory are updated upon reaching acheckpoint, the method of utilizing a checkpoint locations model mayfurther include re-executing the checkpoint locations model andobtaining an updated output of the checkpoint locations model.

According to some embodiments, the method of utilizing a checkpointlocations model may further include adjusting the locations of one ormore subsequent checkpoints based on the updated output of thecheckpoint locations model.

According to some embodiments, the method of utilizing a checkpointlocations model may further include obtaining from a user at least oneof a confirmation to the adjusted locations of the one or moresubsequent checkpoints and an adjustment thereto.

According to some embodiments, the automated medical device may beconfigured to steer the medical instrument toward the target in anon-linear trajectory.

According to some embodiments, there is provided a system for utilizinga checkpoint locations model for optimizing locations of checkpointsalong a trajectory in an image-guided procedure for inserting a medicalinstrument to a target in a body of a patient, the system includes:

an inference module which may include:

-   -   a memory configured to store the one or more new datasets; and    -   one or more processors configured to execute the method of        utilizing a checkpoint locations model as disclosed herein.

According to some embodiments, the inference module may be located on aremote server, an “on premise” server or a computer associated with theautomated medical device.

According to some embodiments, the remote server may be a cloud server.

According to some embodiments, there is provided a computer-readablestorage medium having stored therein machine learning software,executable by one or more processors, for generating a checkpointlocations model for optimizing locations of checkpoints along atrajectory in an image-guided procedure for inserting a medicalinstrument to a target in a body of a patient, by executing the methodsdisclosed herein.

According to some embodiments, there is provided a non-transitorycomputer readable medium storing computer program instructions forgenerating a checkpoint locations model for optimizing locations ofcheckpoints along a trajectory in an image-guided procedure forinserting a medical instrument to a target in a body of a patient, thecomputer program instructions when executed by a processor cause theprocessor to perform operations which may include: collecting one ormore datasets, at least one of the one or more datasets being related toan automated medical device configured to steer a medical instrumenttoward a target in the body of a patient and/or to operation thereof;creating a training set which may include at least a portion of the oneor more datasets and one or more target parameters relating tocheckpoint locations along a trajectory in one or more previousimage-guided procedures for inserting a medical instrument toward atarget in a body of a patient; training the checkpoint locations modelto predict checkpoint locations using the training set; calculating acheckpoint locations prediction error; and optimizing the checkpointlocations model using the calculated checkpoint locations predictionerror.

Certain embodiments of the present disclosure may include some, all, ornone of the above advantages. One or more other technical advantages maybe readily apparent to those skilled in the art from the figures,descriptions, and claims included herein. Moreover, while specificadvantages have been enumerated above, various embodiments may includeall, some, or none of the enumerated advantages.

BRIEF DESCRIPTION OF THE DRAWINGS

Some exemplary implementations of the methods and systems of the presentdisclosure are described with reference to the accompanying drawings. Inthe drawings, like reference numbers indicate identical or substantiallysimilar elements.

FIG. 1 shows a schematic illustration of a system for generating andusing data-analysis model(s)/algorithm(s), according to someembodiments;

FIGS. 2A-2B show perspective views of an exemplary device (FIG. 2A) andan exemplary console (FIG. 2B) of a system for inserting a medicalinstrument toward an internal target, according to some embodiments;

FIG. 3 shows an exemplary trajectory for a medical instrument to reachan internal target within the body of the subject, according to someembodiments;

FIGS. 4A-4D show planning of an exemplary trajectory for inserting andsteering a medical instrument toward a target, on CT images, accordingto some embodiments;

FIGS. 5A-5D show schematic illustrations of real-time updating of atrajectory for inserting and steering a medical instrument toward atarget, according to some embodiments;

FIG. 6 shows a diagram of a method of generating, deploying and using adata-analysis algorithm, according to some embodiments;

FIGS. 7A-7B show an exemplary training module (FIG. 7A) and an exemplarytraining process (FIG. 7B) for training a data-analysis algorithm,according to some embodiments;

FIGS. 8A-8B show an exemplary inference module (FIG. 8A) and anexemplary inference process (FIG. 8B) for utilizing a data-analysisalgorithm, according to some embodiments;

FIG. 9A shows a CT image of a subject illustrating marked recommended“no-fly” zones (i.e., regions that should be avoided during instrumentinsertion), according to some embodiments;

FIG. 9B shows a CT image of a subject demonstrating real-time targetmovement during a needle insertion procedure, according to someembodiments;

FIG. 9C shows a CT image of a subject showing checkpoints (CPs) locatedalong a planned trajectory, according to some embodiments;

FIG. 10 shows a block diagram of exemplary datasets used for generatingan AI model for optimizing checkpoint locations, and exemplary output ofthe checkpoint model, according to some embodiments;

FIG. 11 shows a block diagram illustrating an exemplary method oftraining an AI model for optimizing checkpoint locations, according tosome embodiments;

FIG. 12 shows a flowchart illustrating the steps of a method ofutilizing an AI model for optimizing checkpoint locations along atrajectory, according to some embodiments;

FIG. 13 shows a flowchart illustrating the steps of a method ofutilizing an AI model for creating a “no-fly” zone map, according tosome embodiments;

FIG. 14 shows a block diagram of exemplary datasets used for generatingan AI model for prediction and/or detection of pneumothorax, andexemplary output of the checkpoint model, according to some embodiments;

FIG. 15 shows a block diagram illustrating an exemplary method ofgenerating an AI model for prediction and/or detection of pneumothorax,according to some embodiments;

FIG. 16 shows a flowchart illustrating the steps of a method ofutilizing a pneumothorax model for prediction and/or detection ofpneumothorax, according to some embodiments;

FIG. 17 shows a flowchart illustrating the steps of a method ofutilizing a pneumothorax model for prediction and/or detection ofinternal bleeding, according to some embodiments.

DETAILED DESCRIPTION

The principles, uses and implementations of the teachings herein may bebetter understood with reference to the accompanying description andfigures. Upon perusal of the description and figures present herein, oneskilled in the art will be able to implement the teachings hereinwithout undue effort or experimentation. In the figures, same referencenumerals refer to same parts throughout.

In the following description, various aspects of the invention will bedescribed. For the purpose of explanation, specific details are setforth in order to provide a thorough understanding of the invention.However, it will also be apparent to one skilled in the art that theinvention may be practiced without specific details being presentedherein. Furthermore, well-known features may be omitted or simplified inorder not to obscure the invention.

In some embodiments, there are provided computerized systems and methodsfor generating and using data analysis algorithms and/or AI-basedalgorithms for optimizing various operating parameters of automatedmedical devices and/or providing recommendations to the users ofautomated medical devices and/or predicting clinical conditions (e.g.,complications), based on datasets and parameters derived from or relatedto the operation of the automated medical devices.

In some embodiments, one or more of the generated algorithms may be usedprior to the medical procedure to be performed using the automatedmedical device, e.g., during the planning stage of the procedure. Insome embodiments, one or more of the generated algorithms may be usedduring the medical procedure, e.g., for analyzing in real-time theoperation of the medical device, predicting tissue movement, etc. Insome embodiments, one or more of the generated algorithms may be usedfollowing the medical procedure, e.g., for analyzing the performance ofthe medical device, analyzing the outcome(s) of the procedure, etc.

In some embodiments, one or more of the generated algorithms may be usedto enhance various operating parameters of other medical devices,different from the automated medical device, which may be utilized inthe same medical procedure. For example, some algorithms may provideoperating recommendations and/or instructions relating to parameters ofan imaging system (such as CT, ultrasound, etc.) used in the medicalprocedure. Providing recommendations and/or controlling the operatingparameters of the imaging system may, in some embodiments, allow furtherenhancement of the performance of the automated medical device.

In some embodiments, one or more of the generated algorithms may be usedto enhance various operating parameters of other medical devices,different from the automated medical device, which may be utilized inother medical procedures. Further, one or more of the generatedalgorithms may be used in procedures carried out manually by a user(e.g., physician). For example, an algorithm which can predict theprobability of a medical complication (e.g., pneumothorax) may be usedin manually performed medical procedures (e.g., lung biopsy).

Reference is now made to FIG. 1 , which schematically illustrates asystem 10 for generating and using data-analysis model(s)/algorithm(s),according to some embodiments. As shown in FIG. 1 , various datasets 12are collected from and/or based on the operation of a plurality (N) ofautomated medical devices (shown as Devices 1, 2, 3, . . . n), as wellas on other related datasets (such as, patient related datasets,administrative related datasets, etc.). The datasets 12 may be used forgenerating a variety of specialized data-analysis algorithms/models 14,which may utilize artificial intelligence tools, as detailed below. Thegenerated models/algorithms may then be used for providingrecommendations, operating instructions, enhancements, predictionsand/or alerts 16, for example, to enhance and improve subsequent medicalprocedures 18. In some embodiments, the generation of themodels/algorithms is facilitated using various datasets and/or variousparameters related to or derived from the datasets, to create one ormore training sets, based upon which, the models/algorithms are created,as described in more detail hereinbelow.

In some embodiments, the automated medical device is used for insertionand steering of a medical instrument in a subject's body. In someembodiments, the steering of the medical instrument within the body of asubject may be based on planning and real-time updating the trajectory(2D and/or 3D) of the medical instrument (e.g., of the tip thereof)within the body of the subject, to facilitate the safe and accuratereaching of the tip to an internal target region within the subject'sbody, by the most efficient and safe route.

Reference is now made to FIG. 2A, which shows an exemplary automatedmedical device for inserting a medical instrument in a body of asubject, according to some embodiments. As shown in FIG. 2A, the device20 may include a housing (also referred to as “cover”) 21 accommodatingtherein at least a portion of the steering mechanism. The steeringmechanism may include at least one moveable platform (not shown) and atleast two moveable arms 26A and 26B, configured to allow or controlmovement of an end effector (also referred to as “control head”) 24, atany one of desired movement angles or axis, as disclosed, for example,in abovementioned U.S. Patent Application Publication No. 2019/290,372.The moveable arms 26A and 26B may be configured as piston mechanisms. Tothe end 28 of control head 24, a suitable medical instrument (not shown)may be connected, either directly or by means of a suitable insertionmodule, such as the insertion module disclosed in co-owned U.S. PatentApplication Publication No. 2017/258,489, which is incorporated hereinby reference in its entirety. The medical instrument may be any suitableinstrument capable of being inserted and steered within the body of thesubject, to reach a designated target, wherein the control of theoperation and movement of the medical instrument is effected by thecontrol head 24. The control head 24 may include a driving mechanism(also referred to as “insertion mechanism”) configured to advance themedical instrument toward the target in the patient's body. The controlhead 24 may be controlled by a suitable control system, as detailedherein.

According to some embodiments, the medical instrument may be selectedfrom, but not limited to: a needle, probe (e.g., an ablation probe),port, introducer, catheter (such as a drainage needle catheter),cannula, surgical tool, fluid delivery tool, or any other suitableinsertable tool configured to be inserted into a subject's body fordiagnostic and/or therapeutic purposes. In some embodiments, the medicaltool includes a tip at the distal end thereof (i.e., the end which isinserted into the subject's body).

In some embodiments, the device 20 may have a plurality of degrees offreedom (DOF) in operating and controlling the movement the of themedical instrument along one or more axis. For example, the device mayhave up to six degrees of freedom. For example, the device may have atleast five degrees of freedom. For example, the device may have fivedegrees of freedom, including two linear translation DOF (in a firstaxis), a longitudinal linear translation DOF (in a second axissubstantially perpendicular to the first axis) and two rotational DOF.For example, the device may have forward-backward and left-right lineartranslations facilitated by two moveable platforms, front-back andleft-right rotations facilitated by two moveable arms (e.g., pistonmechanism), and longitudinal translation toward the subject's bodyfacilitated by the insertion mechanism. In some embodiments, the controlsystem (i.e., processor and/or controller) may be capable of controllingthe steering mechanism (including the moveable platforms and themoveable arms) and the insertion mechanism simultaneously, thus enablingnon-linear steering of the medical instrument, i.e., enabling themedical instrument to reach the target by following a non-lineartrajectory. In some embodiments, the device may have six degrees offreedom, including the five degrees of freedom described above and, inaddition, rotation of the medical instrument about its longitudinalaxis. In some embodiments, rotation of the medical instrument about itslongitudinal axis may be facilitated by a designated rotation mechanism.In some embodiments, the control system (i.e., processor and/orcontroller) may be capable of controlling the steering mechanism, theinsertion mechanism and the rotation mechanism simultaneously.

In some embodiments, the device may further include a base 23, whichallows positioning of the device on or in close proximity to thesubject's body. In some embodiments, the device may be configured forattachment to the subject's body either directly or via a suitablemounting surface, such as the mounting base disclosed in co-owned U.S.Patent Application Publication No. 2019/125,397, or the attachmentapparatus disclosed in co-owned International Patent ApplicationPublication No. WO 2019/234,748, both of which are incorporated hereinby reference in their entireties. Attachment of the device 20 to themounting surface may be carried out using dedicated latches, such aslatches 27A and 27B. In some embodiments, the device may be couplable toa dedicated arm or base which is secured to the patient's bed, to a cartpositioned adjacent the patient's bed or to an imaging device (if used),and held on the subject's body or in close proximity thereto, asdescribed, for example, in abovementioned U.S. Pat. No. 10,507,067 andin U.S. Pat. No. 10,639,107, which is incorporated herein by referencein its entirety.

In some embodiments, the device may include electronic components andmotors (not shown) allowing the controlled operation of the device 20 ininserting and steering the medical instrument. In some exemplaryembodiments, the device may include one or more Printed Circuit Board(PCB) (not shown) and electrical cables/wires (not shown) to provideelectrical connection between a controller (not shown) and the motors ofthe device and other electronic components thereof. In some embodiments,the controller may be embedded, at least in part, within device 20. Insome embodiments, the controller may be a separate component. In someembodiments, the device 20 may include a power supply (e.g., one or morebatteries) (not shown). In some embodiments, the device 20 may beconfigured to communicate wirelessly with the controller and/orprocessor. In some embodiments, device 20 may include one or moresensors, such as, but not limited to: a force sensor, an accelerationsensor and/or a radiation sensor (not shown). Use of sensor/s forsensing parameters associated with the interaction between a medicalinstrument and a bodily tissue, e.g., a force sensor, and utilizing thesensor data for monitoring and/or guiding the insertion of theinstrument and/or for initiating imaging, is described, for example, inco-owned U.S. Patent Application Publication No. 2018/250,078, which isincorporated herein by reference in its entirety.

In some embodiments, the housing 21 is configured to cover and protect,at least partially, the mechanical and/or electronic components ofdevice 20 from being damaged or otherwise compromised. In someembodiments, the housing 21 may include at least one adjustable cover,and it may be configured to protect the device from being soiled bydirt, as well as by blood and/or other bodily fluids, thuspreventing/minimizing the risk of cross-contamination between patients,as disclosed, for example, in co-owned International Patent ApplicationNo. PCT/IL2020/051220, which is incorporated herein by reference in itsentirety.

In some embodiments, the device may further include registrationelements disposed at specific locations on the device 20, such asregistration elements 29A and 29B, for registration of the device to theimage space, in image-guided procedures. In some embodiments,registration elements may be disposed on the mounting surface to whichdevice 20 may be coupled, either instead or in addition to registrationelements disposed on device 20. In some embodiments, the device mayinclude a CCD/CMOS camera mounted on the device and/or on the device'sframe and/or as a separate apparatus, allowing the collection of visualimages and/or videos of the patient's body during a medical procedure.

In some embodiments, the medical instrument is configured to beremovably coupleable to the device 20, such that the device can be usedrepeatedly with new medical instruments. In some embodiments, themedical instruments are disposable. In some embodiments, the medicalinstruments are reusable.

In some embodiments, device 20 is part of a system for inserting andsteering a medical instrument in a subject's body based on a preplannedand, optionally, real-time updated trajectory, as disclosed, forexample, in abovementioned co-owned International Application No.PCT/IL2020/051219. In some embodiments, the system may include thesteering and insertion device 20, as disclosed herein, and a controlunit (or—“workstation” or “console”) configured to allow control of theoperating parameters of device 20. In some embodiments, the user mayoperate the device 20 using a pedal or an activation button. In someembodiments, the system may include a remote control unit, which mayenable the user to activate the device 20 from a remote location, suchas the control room adjacent the procedure room (e.g., CT suite), adifferent location at the medical facility or even a location outsidethe medical facility. In some embodiments, the user may operate thedevice using voice commands.

Reference is now made to FIG. 2B, which shows an exemplary workstation(also referred to as “console”) 25 of an insertion system, according tosome embodiments. The workstation 25 may include a display 252 and auser interface (not shown). In some embodiments, the user interface maybe in the form of buttons, switches, keys, keyboard, computer mouse,joystick, touch-sensitive screen, and the like. The monitor and userinterface may be two separate components, or they may form together asingle component (e.g., in the form of a touch-screen). The workstation25 may include one or more suitable processors (for example, in the formof a PC) and one or more suitable controllers, configured to physicallyand/or functionally interact with device 20, to determine and controlthe operation thereof. The one or more processors may be implemented inthe form of a computer (such as a workstation, a server, a PC, a laptop,a tablet, a smartphone or any other processor-based device). In someembodiments, the workstation 25 may be portable (e.g., by having orbeing placed on a movable platform 254).

In some embodiments, the one or more processors may be configured toperform one or more of: determine (plan) a trajectory for the medicalinstrument to reach the target; update the trajectory in real-time, forexample due to movement of the target from its initial identifiedposition as a result of the advancement of the medical instrument withinthe patient's body; present the planned and/or updated trajectory on themonitor 252; control the movement (insertion/steering) of the medicalinstrument based on the planned and/or updated trajectory by providingexecutable instructions (directly or via the one or more controllers) tothe device; determine the actual location of the tip of medicalinstrument by performing required compensation calculations; receive,process and visualize on the monitor images or image-views created froma set of images (between which the user may be able to scroll),operating parameters and the like; or any combination thereof.

In some embodiments, the use of AI-based models (e.g., machine-learningand/or deep-learning based models) requires a “training” stage in whichcollected data is used to create (train) models. The generated (trained)models may later be used for “inference” to obtain specific insights,predictions and/or recommendations when applied to new data during theclinical procedure or at any later time.

In some embodiments, the insertion system and the system creating(training) the algorithms/models may be separate systems (i.e., each ofthe systems includes a different set of processors, memory modules,etc.). In some embodiments, the insertion system and the system creatingthe algorithms/models may be the same system. In some embodiments, theinsertion system and the system creating the algorithms/models may shareone or more resources (such as, processors, memory modules, GUI, and thelike). In some embodiments, the insertion system and the system creatingthe algorithms/models may be physically and/or functionally associated.Each possibility is a separate embodiment.

In some embodiments, the insertion system and the system utilizing thealgorithms/models for inference may be separate systems (i.e., each ofthe systems includes a different set of processors, memory modules,etc.). In some embodiments, the insertion system and the systemutilizing the algorithms/models for inference may be the same system. Insome embodiments, the insertion system and the system utilizing thealgorithms/models for inference may share one or more resources (suchas, processors, memory modules, GUI, and the like). In some embodiments,the insertion system and the system utilizing the algorithms/models forinference may be physically and/or functionally associated. Eachpossibility is a separate embodiment.

In some embodiments, the device may be configured to operate inconjunction with an imaging system, including, but not limited to:X-Ray, CT, cone beam CT, CT fluoroscopy, Mill, ultrasound, or any othersuitable imaging modality. In some embodiments, the steering of themedical instrument based on a planned and, optionally, real-time updated2D or 3D trajectory of the tip of the medical instrument, may beimage-guided.

According to some embodiments, during the operation of the automatedmedical device, various types of data may be generated, accumulatedand/or collected, for further use and/or manipulation, as detailedbelow. In some embodiments, the data may be divided into varioustypes/sets of data, including, for example, data related to operatingparameters of the device, data related to clinical procedures, datarelated to the treated patient, data related to administrativeinformation, and the like, or any combination thereof.

In some embodiments, such collected datasets may be collected from oneor more (i.e., a plurality) of automated medical devices, operatingunder various circumstances (for example, different procedures,different medical instruments, different patients, different locationsand operating staff, etc.), to thereby generate a large data base (“bigdata”), that can be used, utilizing suitable data analysis tools and/orAI-based tools to ultimately generate models or algorithms that allowperformance enhancements, automatic control or affecting control (i.e.,by providing recommendations), of the medical devices. Thus, bygenerating such advantageous and specialized models or algorithms,enhanced control and/or operation of the medical device may be achieved.

Reference is now made to FIG. 3 , which schematically shows a trajectoryplanned using a processor, such as the processor(s) of the insertionsystem described in FIG. 2B, for delivering a medical instrument to aninternal target within the body of the subject, using an automatedmedical device, such as the automated device of FIG. 2A. In someembodiments, the planned trajectory may be linear or substantiallylinear. In some embodiments, and as shown in FIG. 3 , the trajectory maybe non-linear trajectory having any suitable/acceptable degree ofcurvature.

In some embodiments, the one or more processors may calculate a plannedtrajectory for the medical instrument to reach the target. The planningof the trajectory and the controlled steering of the instrumentaccording to the planned trajectory may be based on a model of themedical instrument as a flexible beam having a plurality of virtualsprings connected laterally thereto to simulate lateral forces exertedby the tissue on the instrument, thereby calculating the trajectorythrough the tissue on the basis of the influence of the plurality ofvirtual springs on the instrument, and utilizing an inverse kinematicssolution applied to the virtual springs model to calculate the requiredmotion to be imparted to the instrument to follow the plannedtrajectory. The processor may then provide motion commands to theautomated device, for example via a controller. In some embodiments,steering of the medical instrument may be controlled in a closed-loopmanner, whereby the processor generates motion commands to the automateddevice and receives feedback regarding the real-time location of themedical instrument (e.g., the tip thereof), which is then used forreal-time trajectory corrections, as disclosed, for example, inabovementioned U.S. Pat. No. 8,348,861. For example, if the instrumenthas deviated from the planned trajectory, the processor may calculatethe motion to be applied to the robot to reduce the deviation. Thereal-time location of the medical instrument and/or the corrections maybe calculated and/or applied using data-analysis models/algorithms. Insome embodiments, certain deviations of the medical instrument from theplanned trajectory, for example deviations which exceed a predeterminedthreshold, may require recalculation of the trajectory for the remainderof the procedure, as described in further detail hereinbelow.

As shown in FIG. 3 , a trajectory 32 is planned between an entry point36 and an internal target 38. The planning of the trajectory 32 may takeinto account various variables, including, but not limited to: the typeof the medical instrument to be used and its characteristics, thedimensions of the medical instrument (e.g., length, gauge), the type ofimaging modality (such as, CT, CBCT, MRI, X-Ray, CT fluoroscopy,ultrasound and the like), the tissues through which the medicalinstrument is to be inserted, the location of the target, the size ofthe target, the insertion point, the angle of insertion (relative to oneor more axis), milestone points (“secondary targets” through which themedical instrument should pass) and the like, or any combinationthereof. In some embodiments, at least one of the milestone points maybe a pivot point, i.e., a predefined point along the trajectory in whichthe deflection of the medical instrument is prevented or minimized, tomaintain minimal pressure on the tissue (even if this results in alarger deflection of the instrument in other parts of the trajectory).In some embodiments, the planned trajectory is an optimal trajectorybased on one or more of these parameters. Further taken into account indetermining the trajectory may be various obstacles 39A-39C, which maybe identified along the path and which should be avoided, to preventdamage to the neighboring tissues and/or to the medical instrument.According to some embodiments, safety margins 34 may be marked along theplanned trajectory 32, to ensure a minimal distance between thetrajectory 32 and potential obstacles en route. The width of the safetymargins may be symmetrical in relation to the trajectory 32. The widthof the safety margins may be asymmetrical in relation to the trajectory32. According to some embodiments, the width of the safety margins 34may be preprogrammed. According to some embodiments, the width of thesafety margins may be automatically set, or recommended to the user, bythe processor, based on data obtained from previous procedures using adata analysis algorithm. According to some embodiments, the width of thesafety margins 34 may be determined and/or adjusted by the user. Furthershown in FIG. 3 is an end of a control head 30 of the exemplaryautomated insertion device, to which the medical instrument (not shownin FIG. 3 ) is coupled, as virtually displayed on the monitor, toindicate its position and orientation.

The trajectory 32 shown in FIG. 3 is a planar trajectory (i.e., twodimensional). In some embodiments, steering of the instrument is carriedout according to a planner trajectory, for example trajectory 32. Insome embodiments, the calculated planner trajectory may besuperpositioned with one or more additional planner trajectories, toform a three-dimensional (3D) trajectory. Such additional plannertrajectories may be planned on one or more different planes, which maybe perpendicular to the plane of the first planner trajectory (e.g.,trajectory 32) or otherwise angled relative thereto. According to someembodiments, the 3D trajectory may include any type of trajectory,including a linear trajectory or a non-linear trajectory.

According to some embodiments, the steering of the medical instrument iscarried out in a 3D space, wherein the steering instructions aredetermined on each of the planes of the superpositioned plannertrajectories, and are then superpositioned to form the steering in thethree-dimensional space. The data/parameters/values thus obtained duringthe steering of the medical instrument using the automated device can beused as data/parameters/values for the generation/training and/orutilization/inference of the data-analysis model(s)/algorithm(s).

Reference is now made to FIGS. 4A-4D, which show planning of anexemplary trajectory for inserting and steering a medical instrumenttoward a target, according to some embodiments. The exemplary trajectorymay be planned using a processor, such as the processor(s) of theinsertion system described in FIG. 2B, and the insertion and steering ofthe medical instrument toward the target according to the plannedtrajectory may be executed using an automated insertion device, such asthe automated device of FIG. 2A.

The planning in FIGS. 4A-4D is shown on CT image-views, however it canbe appreciated that the planning can be carried out similarly on imagesobtained from other imaging systems, such as ultrasound, Mill and thelike. Shown in FIG. 4A are CT image-views of a subject, depicting at theleft-hand panel an axial plane view and on the right-hand panel asagittal plane view. Also indicated in the figure is an internal target44 and an automated insertion device 40. Further indicated is a vertebra46. In FIG. 4B, which shows the CT image-views of FIG. 4A, the insertionpoint 42 is indicated. Consequently, according to some embodiments, alinear trajectory 48 between the insertion point 42 and the internaltarget 44 may be calculated and displayed on each of the two views (forexample, axial plane view and sagittal plane view). Typically, a lineartrajectory is preferred, thus, if the displayed linear trajectory doesnot pass in close proximity to any potential obstacles, then the lineartrajectory is determined as the planned trajectory for the insertionprocedure. In FIG. 4C, a transverse process 462 of vertebra 46 isdetected in close proximity to the calculated linear trajectory, and isidentified and marked, in this example on the axial plane view, to allowconsidering the obstacle when planning the trajectory for the procedure.In FIG. 4D, the trajectory is re-calculated, so as to allow theinstrument to avoid contacting the obstacle 462, resulting in anon-linear trajectory 48′. According to some embodiments, the plannedtrajectory may not be calculated until potential obstacles are marked onthe image-view/s, either manually or automatically, until the userconfirms that there are no potential obstacles and/or until the usermanually initiates trajectory calculation. In such embodiments, if thereare obstacles which necessitate a non-linear trajectory, an interimlinear trajectory, similar to linear trajectory 48 of FIG. 4B, may notbe calculated and/or displayed. According to some embodiments, a maximalallowable curvature level may be pre-set for the calculation of thenon-linear trajectory. The maximal curvature threshold may depend, forexample, on the trajectory parameters (e.g., distance between the entrypoint and the target) and on the type of instrument intended to be usedin the procedure and its characteristics (for example, type, diameter(gauge), and the like). As further detailed below, the plannedtrajectory may be updated in real-time based on the real-time positionof the medical instrument (for example, the tip thereof) and/or thereal-time position of the target and/or the real-time positions ofobstacle/s.

According to some embodiments, the target 44, insertion point 42 and,optionally, obstacle/s, such as transverse process 462, are markedmanually by the user. According to other embodiments, the processor ofthe insertion system (or of a separate system) may be configured toidentify and mark at least one of the target, the insertion point andthe obstacle/s, and the user may, optionally, be prompted to confirm oradjust the processor's proposed markings. In such embodiments, thetarget and/or obstacle/s may be identified using known image processingtechniques and/or data-analysis models/algorithms, based on dataobtained from previous procedures. The insertion point may be suggestedbased solely on the obtained images, or, alternatively or additionally,on data obtained from previous procedures using data-analysismodels/algorithms.

According to some embodiments, the trajectory may be calculated basedsolely on the obtained images and the marked locations of the entrypoint, target (and, optionally, obstacle/s). According to otherembodiments, the calculation of the trajectory may be based also on dataobtained from previous procedures, using data-analysismodels/algorithms. According to some embodiments, once the plannedtrajectory has been determined, checkpoints along the trajectory may beset. The checkpoints may be manually set by the user, or they may beautomatically set or recommended by the processor, as described infurther detail hereinbelow.

It can be appreciated that although axial and sagittal views are shownin FIGS. 4A-4D, views pertaining to different planes or orientations(e.g., coronal, pseudo axial, pseudo sagittal, pseudo coronal, etc.) oradditionally generated views (e.g., trajectory view, tool view, 3D view,etc.), may be used in order to perform and/or display the trajectoryplanning.

Reference is now made to FIGS. 5A-5D, which show schematic illustrationsof real-time updating of a trajectory for inserting and steering amedical instrument toward a target, according to some embodiments. Thetrajectory may be updated using a processor, such as the processor(s) ofthe insertion system described in FIG. 2B, and the insertion andsteering of the medical instrument toward the target according to theplanned and updated trajectories may be executed using an automatedinsertion device, such as an automated device 50. In some embodiments,the automated device 50 may be body-mountable, for example, as shown inFIGS. 5A-5D, the device 50 may be configured for attachment to asubject's body using an attachment apparatus 52, such as the attachmentapparatus described in abovementioned co-owned International PatentApplication Publication No. WO 2019/234,748.

According to some embodiments, once the planned trajectory has beendetermined, checkpoints along the trajectory may be set. Checkpoints maybe used to pause the insertion of the medical instrument and initiateimaging of the region of interest, to verify the position of theinstrument (specifically, in order to verify that the instrument (e.g.,the tip thereof) follows the planned trajectory), to monitor thelocation of the marked obstacles and/or identify previously unmarkedobstacles along the trajectory, and to verify the target's position,such that recalculation of the trajectory may be initiated, if the userchooses to do so, before advancing the instrument to the nextcheckpoint/the target. The checkpoints may be manually set by the user,or they may be automatically set or recommended by the processor, asdescribed in further detail hereinbelow. According to some embodiments,the checkpoints may be positioned at a spatial-pattern, atemporal-pattern, or both. According to some embodiments, thecheckpoints may be reached at predetermined time intervals, for example,every 2-5 seconds. According to some embodiments, the checkpoints may bespaced apart, including the first checkpoint from the entry point andthe last checkpoint from the target organ and/or target point, at anessentially similar distance along the trajectory, for example every20-50 mm. According to some embodiments, upper and/or lower intervalthresholds between checkpoints may be predetermined. For example, thecheckpoints may be automatically set by the processor at default 20 mmintervals, and the user can then adjust the distance between each twocheckpoints (or between the entry point and the first checkpoint and/orbetween the last checkpoint and the target) such that the maximaldistance between them is 30 mm and/or the minimal distance between themis 3 mm, for example.

The trade-off of utilizing many checkpoints is prolonged procedure time,as well as repeated exposure to radiation. On the other hand, too littlecheckpoints may affect the accuracy and safety of the medical procedure.In the example shown in FIGS. 5A-5D, three checkpoints have been setalong the trajectory.

FIG. 5A shows a medical instrument 54 being inserted toward a target 505in the subject's body and reaching the first checkpoint 512, accordingto a preplanned trajectory 510. In some embodiments, the preplannedtrajectory 510 is a linear or substantially linear trajectory. FIG. 5Bshows the medical instrument 54 being further inserted into thesubject's body, reaching the third checkpoint 514 along the plannedtrajectory 510. As shown in FIG. 5B, the target 505 has moved from itsinitial position during and as a result of the advancement of themedical instrument within the tissue. In some embodiments, thedetermination of the real-time location of the target may be performedmanually by the user, i.e., the user visually identifies the target inimages (continuous or manually or automatically initiated, for examplewhen the instrument reaches a checkpoint), and marks the new targetposition on the GUI. In some embodiments, the determination of thereal-time target location may be performed automatically by a processorusing image processing techniques and/or data-analysis algorithm(s). Insome embodiments, once it has been determined that the real-timelocation of the target deviates from its initial location, i.e., thatthe target has moved, the deviation may be compared to a predeterminedthreshold to determine if the deviation exceeds the threshold. Thethreshold may be, for example, a set value or a percentage reflecting achange in a value. The threshold may be determined by the user or it maybe determined by the processor, for example using a data-analysisalgorithm based on data collected in previous procedures. In someembodiments, if the deviation does not exceed the predeterminedthreshold, it may be decided, either by the user or automatically by theprocessor, that the insertion procedure may continue based on thepreplanned trajectory. If the deviation exceeds the predeterminedthreshold, then it may be decided, either by the user or automaticallyby the processor, that recalculation of the trajectory is required.

According to some embodiments, recalculation of the trajectory may alsobe required if the instrument deviated from the planned trajectory abovea predetermined deviation threshold. In some embodiments, determiningthe actual real-time location of the instrument may require applying acorrection to the determined location of the tip of the medicalinstrument, to compensate for deviations due to imaging artifacts. Theactual location of the tip may be determined based on an instrumentposition compensation “look-up” table, which corresponds to the imagingmodality and the medical instrument used, as disclosed, for example, inabovementioned co-owned International Patent Application No.PCT/IL2020/051219. In some embodiments, if the real-time location of themedical instrument indicates that the instrument has deviated from theplanned trajectory, but the deviation does not exceed the predetermineddeviation threshold, one or more checkpoints may be added and/orrepositioned along the planned trajectory, either manually by the useror automatically by the processor, to direct the instrument back to theplanned trajectory. In some embodiments, the processor may prompt theuser to add and/or reposition checkpoint/s. In some embodiments, theprocessor may recommend to the user specific position/s for the newand/or repositioned checkpoints. Such a recommendation may be generatedusing data-analysis algorithm(s).

According to some embodiments, recalculation of the trajectory may alsobe required if, for example, an obstacle is identified along thetrajectory. Such an obstacle may be an obstacle which was marked(manually or automatically) prior to the calculation of the plannedtrajectory but tissue movement, e.g., tissue movement resulting from theadvancement of the instrument within the tissue, caused the obstacle tomove such that it entered the planned path. In some embodiments, theobstacle may be a new obstacle, i.e., an obstacle which was not visiblein the image (or set of images) based upon which the planned trajectorywas calculated, and became visible during the insertion procedure.

In some embodiments, if the instrument deviated from the plannedtrajectory (e.g., above a predetermined deviation threshold), a new orrepositioned obstacle is identified along the planned trajectory and/orthe target has moved (e.g., above a predetermined threshold), the usermay be prompted to initiate an update (recalculation) of the trajectory.In some embodiments, recalculation of the trajectory, if required, isexecuted automatically by the processor and the insertion of theinstrument is automatically resumed based on the updated trajectory. Insome embodiments, recalculation of the trajectory, if required, isexecuted automatically by the processor, however the user is prompted toconfirm the recalculated trajectory before advancement of the instrument(e.g., to the next checkpoint) according to the updated trajectory canbe resumed.

As shown in FIG. 5C, the trajectory has been recalculated based on thenew determined location of the target 505, resulting in an updatedtrajectory 510′. In some embodiments, the updated trajectory 510′ is aplanner (2D) trajectory. In some embodiments, the updated trajectory510′ is a three-dimensional trajectory, which is calculated by firstcalculating two or more planner trajectories and then superpositioningthe two or more planner trajectories to form the updated 3D trajectory.FIG. 5D shows the medical instrument 54 reaching the target at its newlocation, after following the updated trajectory 510′. As shown,although the preplanned trajectory 510 was linear, the recalculation ofthe trajectory due to movement of the target 505, resulted in themedical instrument 54, specifically the tip of the instrument, followinga non-linear trajectory to accurately reach the target.

Reference is now made to FIG. 6 , which is a diagram 60 of a method ofgenerating, deploying and using a data-analysis algorithm, according tosome embodiments. As shown in FIG. 6 , at step 61, automated medicalprocedure(s) are executed using automated medical device(s). Automatedmedical procedure(s) involve a plurality of datasets related thereto (asfurther detailed below). For example, some of the datasets directlyrelate to the operation of the medical device (such as operatingparameters), some of the datasets relate to the clinical procedure, someof the datasets relate to the treated patient and some of the datasetsrelate to administrative related information. In some embodiments, inaddition to the datasets related or generated during the medicalprocedure/s, datasets may be generated during training sessionsperformed by users on a dedicated simulator system. Such a simulatorsystem may be configured to at least partially simulate a medicalprocedure, including enabling users to plan the procedure on existingimages and then simulating the execution of the procedure according tothe procedure plan via a virtual automated medical device and a virtualmedical instrument. Next, at step 62, at least some of the generateddatasets, values thereof and/or parameters related thereto are collectedfrom the medical procedures and/or simulation sessions and stored in acentralized database. The collected datasets may be split/divided foruse as training sets, validation sets and/or testing sets. Then, at step63, the collected data is annotated, to thereby generate and train thedata-analysis algorithm, at stage 64. At step 65, the data-analysisalgorithm is validated and deployed. Once deployed, the results from thealgorithm are obtained, at step 66, and the results are then used toprovide, at stage 67, recommendations/operatinginstructions/predictions/alerts. Sub sequent medical procedures executedby automated medical devices may implement at least some of therecommendations/operating instructions/predictions/alerts, therebyreturning to step 61 and repeating the method. In some instances, theperformance of the validated algorithm is monitored, at stage 68, and isfurther enhanced/improved, based on data stored in the centralizeddatabase and/or on newly acquired data.

According to some embodiments, the various obtained datasets may be usedfor the training, construction and/or validation of the algorithm. Insome embodiments, the datasets may be selected from, but not limited to:medical device related dataset, clinical procedures related dataset,patient related dataset, administrative-related dataset, and the like,or any combination thereof.

According to some exemplary embodiments, the medical device relateddataset may include such data parameters or values as, but not limitedto: procedure steps timing, overall procedure time, overall steeringtime (of the medical instrument), entry point of the medical instrument,target point/regions, target updates (for example, updating real-timedepth and/or lateral position of the target), planned trajectory of themedical instrument, real-time trajectory of the medical instrument,(real-time) trajectory updates, number of checkpoints (CPs) along theplanned or real-time-updated trajectory of the medical instrument, CPpositions/locations, CP updates during the procedure, CP errors (in 2Dand/or in 3D), position of the medical device, insertion angles of themedical instrument (for example, insertion angle in the axial plane andoff-axial angle), indication whether the planned (indicated) target hasbeen reached during the procedure, target error (for example, lateraland depth, in 2D and/or in 3D), scans/images, parameters per scan,radiation dose per scan, total radiation dose in the steering phase ofthe medical instrument, total radiation dose the entire procedure,errors/warnings indicated during the procedure, software logs, motioncontrol traces, medical device registration logs, medical instrument(such as, needle) detection logs, homing and BIT results, and the like,or any combination thereof. Each possibility is a separate embodiment.

In some embodiments, one or more of the values may be configured to becollected automatically by the system. For example, values such asprocedure steps timing, overall steering time, entry, target, targetupdates (depth and lateral), trajectory, trajectory updates, number ofCPs, CP positions, CP updates, CP errors (2 planes and/or 3D), robotposition, scans/images, parameters per scan, errors/warnings, softwarelogs, motion control traces, medical device registration logs, medicalinstrument detection logs, homing and BIT results may be collectedautomatically.

According to some exemplary embodiments, the clinical procedures relateddataset may include such data parameters or values as, but not limitedto: procedure type (e.g., blood/fluid sampling, regional anesthesia,tissue biopsy, catheter insertion, cryogenic ablation, electrolyticablation, brachytherapy, neurosurgery, deep brain stimulation, variousminimally invasive surgeries, and the like), target organ, target size,target type (tumor, abscess, and the like), type of medical instrument,size of medical instrument, complications before/during/after theprocedure, adverse events before/during/after the procedure, respirationsignals of the patient, and the like, or any combination thereof. Eachpossibility is a separate embodiment. In some embodiments, one or moreof the values may be configured to be collected automatically. Forexample, the type of medical instrument (for example, type of a needle),size of the medical instrument (for example, size (gauge) of theneedle), respiration signal(s) of the patient, movement traces of theautomated medical device and system logs may be collected automatically.In some embodiments, one or more of the values may be configured to becollected manually by requesting the user to insert the data,information and/or visual marking using a graphic-user-interface (GUI),for example.

According to some exemplary embodiments, the patient related dataset mayinclude such data parameters or values as, but not limited to: age,gender, race, relevant medical history, vital signs before/after/duringthe procedure, body dimensions (height, weight, BMI, circumference,etc.), current medical condition, pregnancy, smoking habits, demographicdata, and the like, or any combination thereof. Each possibility is aseparate embodiment.

According to some exemplary embodiments, the administrative relateddataset may include such data parameters or values as, but not limitedto: institution (healthcare facility) in which the procedure isperformed, physician, staff, system serial numbers, disposables used,software/operating systems versions, configuration parameters, and thelike, or any combination thereof. Each possibility is a separateembodiment.

According to some embodiments, by using one or more values of one ormore datasets, and generating a data-analysis algorithm, variouspredictions, recommendations and/or implementations may be generatedthat can enhance further medical procedures. In some embodiments, basedon the data used, the generated algorithm/s may be customized to aspecific procedure, specific patient (or cohort of patients), or anyother set of specific parameters.

According to some embodiments, the algorithm/s may be used for enhancingmedical procedures, predicting clinical outcome and/or clinicalcomplications and overall increasing safety and accuracy.

According to some exemplary embodiments, the data-analysis algorithmsgenerated by the systems and methods disclosed herein may be used for,but not limited to: Predicting, prevention and/or detecting variousclinical conditions and/or complications (e.g., pneumothorax, internalbleeding, breathing abnormalities, etc.); Determining or recommendingentry point location; Determining or recommending an optimal trajectoryfor the insertion procedure; Optimizing checkpoint positioning along atrajectory (planned and/or updated trajectory), e.g., by recommendingthe best tradeoff between accuracy and radiation exposure/proceduretime; Determining or recommending “no-fly” zones, i.e., areas (obstaclesand/or vital anatomical structures) to avoid during instrumentinsertion; Predicting and/or detecting entrance into defined “no-fly”zones; Predicting real-time tissue (including target) movement;Automatic (real-time) target tracking; Automatic steering of theinstrument based on real-time target tracking; Optimizing automaticbreathing synchronization; Optimizing the positioning of the medicaldevice relative to a subject's body and/or recommending to the user howto position the medical device relative to the subject's body, asdisclosed, for example, in co-owned International Application No.PCT/IL2020/051247, which is incorporated herein by reference in itsentirety; Optimizing steering algorithm corrections; Optimizing medicaldevice registration and instrument detection algorithms therebyimproving system accuracy and allowing radiation reduction; Optimizingcompensation calculations for determining the actual real-time locationof the tip of the medical instrument, as disclosed, for example, inabovementioned co-owned International Application No. PCT/IL2020/051219;Recommending the medical instrument to be used in the procedure(instrument type, instrument gauge, etc.); Evaluating procedure success(estimated success and/or estimated risk level) based on the currentplanning and similar past procedures; Correlating procedure successand/or morbidity/mortality with different parameters, such as targettype, target size, trajectory, etc.; Minimizing radiation level;Improving image quality (e.g., in case of low-quality imaging system orlow-dose scanning); 3D reconstruction and segmentation of organs andtissues; Integrating obtained images with the subject's medical recordsto fine tune the procedure planning and/or better evaluate risks;Utilizing force sensor measurements for evaluation of tissue compliance,early detection of clinical complications and/or optimizing instrumentsteering; Utilization of additional sensor measurements (e.g.,accelerometer, radiation sensor, etc.); Generating voice commands tooperate the automated device; Use of augmented reality (AR) and/orvirtual reality (VR) for device positioning, target tracking and/orinstrument tracking, etc.; Evaluating clinical procedure efficiency,e.g., evaluating the impact of ablation on the target and thesurrounding tissue (and recommending the ablation treatment areaaccordingly), evaluating drug delivery (including anesthesia) efficiencybased on instrument location and/or volume analysis; Analyzing theoutcome of the procedure, both short term and long term, to identifylong term implications and correlations; Providing data and analysis to,for example, healthcare providers, healthcare facilities, imagingsystems' manufacturers, medical instruments' manufacturers, to be usedas needed; Predicting and/or detecting system failures and ‘servicerequired’ alerts; Medical personnel training programs based on experts'procedures; Medical personnel performance analysis; and the like, or anycombination thereof. Each possibility is a separate embodiment.

According to some embodiments, data-analysis algorithms generated by thesystems and methods disclosed herein may be used for providingprediction, prevention and/or early detection of various clinicalconditions/complications, such as pneumothorax, local bleeding, etc.According to some embodiments, generated algorithms may be used forproviding recommendations regarding various device functions andoperations, including providing optimized routes or modes of operation.According to some embodiments, generated algorithms may be used forproviding improved/optimized procedures, while taking into accountvarious variables that may change during the procedure, such as, forexample, predicting target movement, correlating body movement(breathing-related) and device operation, etc. In some embodiments,generated algorithms may be used to predict service calls and potentialsystem malfunctions. In some embodiments, generated algorithms may beused to allow performance analysis and user feedback, to improve the useof the medical device.

According to some embodiments, a training module (also referred to as“learning module”) may be used to train an AI model (e.g., ML orDL-based model) to be used in an inference module, based on the datasetsand/or the features extracted therefrom and/or additional metadata, inthe form of annotations (e.g., labels, bounding-boxes, segmentationmaps, visual locations markings, etc.). In some embodiments, thetraining module may constitute part of the inference module or it may bea separate module. In some embodiments, a training process (step) mayprecede the inference process (step). In some embodiments, the trainingprocess may be on-going and may be used to update/validate/enhance theinference step (see “active-learning” approach described herein). Insome embodiments, the inference module and/or the training module may belocated on a local server (“on premise”), a remote server (such as, aserver farm or a cloud-based server) or on a computer associated withthe automated medical device. According to some embodiments, thetraining module and the inference module may be implemented usingseparate computational resources. In some embodiments, the trainingmodule may be located on a server (local or remote) and the inferencemodule may be located on a local computational resource (computer), orvice versa. According to some embodiments, both the training module andthe inference module may be implemented using common computationalresources, i.e., processors and memory components shared therebetween.In some embodiments, the inference module and/or the training module maybe located or associated with a controller (or steering system) of anautomated medical device. In such embodiments, a plurality of inferencemodules and/or learning modules (each associated with a medical deviceor a group of medical devices), may interact to share informationtherebetween, for example, utilizing a communication network. In someembodiments, the model(s) may be updated periodically (for example,every 1-36 weeks, every 1-12 months, etc.). In some embodiments, themodel(s) may be updated based on other business logic. In someembodiments, the processor(s) of the automated medical device (e.g., theprocessor of the insertion system) may run/execute the model(s) locally,including updating and/or enhancing the model(s).

According to some embodiments, during training of the model (as detailedbelow), the learning module (either implemented as a separate module oras a portion of the inference module), may be used to construct asuitable algorithm (such as, a classification algorithm), byestablishing relations/connections/patterns/correspondences/correlationsbetween one or more variables of the primary datasets and/or betweenparameters derived therefrom. In some embodiments, the learning may besupervised learning (e.g., classification, object detection,segmentation and the like). In some embodiments, the learning may beunsupervised learning (e.g., clustering, anomaly detection,dimensionality reduction and the like). In some embodiments the learningmay be reinforcement learning. In some embodiments, the learning may usea self-learning approach. In some embodiments, the learning process isautomatic. In some embodiments, the learning process is semi-automatic.In some embodiments, the learning is manually supervised. In someembodiments, at least some variables of the learning process may bemanually supervised/confirmed, for example, by a user (such as aphysician). In some embodiments, the training stage may be an offlineprocess, during which a database of annotated training data is assembledand used for the creation of data-analysis model(s)/algorithm(s), whichmay then be used in the inference stage. In some embodiments, thetraining stage may be performed “online”, as detailed herein.

According to some embodiments, the generated algorithm may essentiallyconstitute at least any suitable specialized software (including, forexample, but not limited to: image recognition and analysis software,statistical analysis software, regression algorithms (linear,non-linear, or logistic etc.), and the like). According to someembodiments, the generated algorithm may be implemented using anartificial neural network (ANN), such as a deep neural network (DNN), aconvolutional neural network (CNN), a recurrent neural network (RNN) andthe like, decision trees or graphs, association rule learning, supportvector machines, inductive logic programming, Bayesian networks,instance-based learning, manifold learning, sub-space learning, and thelike, or any combination thereof. The algorithm or model may begenerated using machine learning tools, data wrangling tools, deeplearning tools, and, more generally, data science and artificialintelligence (AI) learning tools, as elaborated hereinbelow.

Reference is now made to FIGS. 7A-7B, which show an exemplary trainingmodule (FIG. 7A) and an exemplary training process (FIG. 7B), accordingto some embodiments.

As shown in FIG. 7A, a training module 70 may include two main hardwarecomponents/units: at least one memory 72 and at least one processingunit 74, which are functionally and/or physically associated. Trainingmodule 70 may be configured to train a model based on data. Memory 72may include any type of accessible memory (volatile and/ornon-volatile), configured to receive, store and/or provide various typesof data, to be processed by processing unit 74, which may include anytype of at least one suitable processor, as detailed below. In someembodiments, the memory and the processing units may be functionally orphysically integrated, for example, in the form of a Static RandomAccess Memory (SRAM) array. In some embodiments, the memory is anon-volatile memory having stored therein executable instructions (forexample, in the form of a code, service, executable program and/or amodel file). As shown in FIG. 7A, the memory unit 72 may be configuredto receive, store and/or provide various types of data values orparameters related to the data. Memory 72 may store or accept raw(primary) data 722 that has been collected, as detailed herein.Additionally, metadata 724, related to the raw data 722 may also becollected/stored in memory 72. Such metadata may include a variety ofparameters/values related to the raw data, such as, but not limited to:the specific device which was used to provide the data, the time thedata was obtained, the place the data was obtained (such as a specificprocedure/operating room, specific institution, etc.), and the like.Memory 72 may further be configured to store/collect data annotations(e.g., labels) 726. In some embodiments, the collected data may requireadditional steps for the generation of data-annotations that will beused for the generation of the machine-learning, deep-learning models orother statistical or predictive algorithms as disclosed herein. In someembodiments, such data annotations may include labels describing theclinical procedure's characteristics, the automated device's operationand computer-vision related annotations, such as segmentation masks,target marking, organs and tissues marking, and the like. The differentannotations may be generated in an “online” manner, which is performedwhile the data is being collected, or in an “offline” manner, which isperformed at a later time after sufficient data has been collected. Thememory 72 may further include features database 728. The featuresdatabase 728 may include a database (“store”) of previously known orgenerated features that may be used in the training/generation of themodels. The memory 72 of training module 70 may further, optionally,include pre-trained models 729. The pre-trained models 729 includeexisting pre-trained algorithms which may be used to automaticallyannotate a portion of the data and/or to ease training of new modelsusing “transfer-learning” methods and/or to shorten training time byusing the pre-trained models as starting points for the training processon new data and/or to evaluate and compare performance metrics ofexisting versus newly developed models before deployment of new model toproduction, as detailed hereinbelow. In some embodiments, processingunit 72 of training module 70 may include at least one processor,configured to process the data and allow/provide model training byvarious processing steps (detailed in FIG. 7B). Thus, as shown in FIG.7A, processing unit 74 may be configured at least to performpre-processing of the data 742. Pre-processing of the data may includeactions for preparing the data stored in memory 72 for downstreamprocessing, such as, but not limited to, checking for and handling nullvalues, imputation, standardization, handling categorical variables,one-hot encoding, resampling, scaling, filtering, outlier removal etc.Processing unit 74 may further, optionally, be configured to performfeature extraction 744, in order to reduce the raw data dimension and/oradd informative domain-knowledge into the training process and allow theuse of additional machine-learning algorithms not suitable for trainingon raw data and/or optimization of existing or new models by trainingthem on both the raw data and the extracted features. Feature extractionmay be executed using dimensionality reduction methods, for example,Principal Components Analysis (PCA), Independent Component Analysis(ICA), Linear Discriminant Analysis (LDA), Locally Linear Embedding(LLE), t-distributed Stochastic Neighbor Embedding (t-SNE), UnifiedManifold Approximation and Projection (UMAP) and/or Autoencoders, etc.Feature extraction may be executed using feature engineering methods inwhich mathematical tools are used to extract domain-knowledge featuresfrom the raw data, for example—statistical features, such as mean,variance, ratio, frequency etc. and/or visual features, such asdimension or shape of certain objects in an image. Another optionaltechnique which may be executed by the processing unit 74 to reduce thenumber of features in the dataset is feature selection, in which theimportance of the existing features in the dataset is ranked and theless important features are discarded (i.e., no new features arecreated). Processing unit 74 may further be configured to execute modeltraining 746.

Reference is now made to FIG. 7B, which shows steps in an exemplarytraining process 76, executed by a suitable training module (such astraining module 70 of FIG. 7A). As shown in FIG. 7B, at optional step761, the collected datasets may first require an Extract-Transform-Load(ETL) or ELT process that may be used to (1) Extract the data from asingle or multiple data sources (including, but not limited to, theautomated medical device itself, Picture Archiving and CommunicationSystem (PACS), Radiology Information System (RIS), imaging device,healthcare facility's Electronic Health Record (EHR) system, etc.), (2)Transform the data by applying one or more of the following steps:handling missing values, checking for duplicates, converting data typesas needed, encoding values, joining data from multiple sources,aggregating data, translating coded values etc. and (3) Load the data toa variety of data storage devices (on-premise or at a remote location(such as a cloud server)) and/or to a variety of data stores, such asfile systems, SQL databases, no-SQL databases, distributed databases,object storage, etc. In some embodiments, the ETL process may beautomatic and triggered with every new data collected. In otherembodiments, the ETL process may be triggered at a predefined schedule,such as once a day or once a week, for example. In some embodiments,another business logic may be used to decide when to trigger the ETLprocess.

At step 762, the data may be cleaned to ensure high quality data by, forexample removal of duplicates, removal or modification of incorrectand/or incomplete and/or irrelevant data samples, etc. At step 763, thedata is annotated. The data annotations may include, for example, labelsdescribing the clinical procedure's characteristics, the automateddevice's operation and computer-vision related annotations, such assegmentation masks, target marking, organs and tissues marking,existence of medical conditions/complications, existence of certainpathologies, etc. The different annotations may be generated in an“online” manner, which is performed while the data is being collected,or in an “offline” manner, which is performed at a later time aftersufficient data has been collected. In some embodiments, the dataannotations may be generated automatically using an “active learning”approach, in which existing pre-trained algorithms are used toautomatically annotate a portion of the data. In some embodiments, thedata annotations may be generated using a partially automated approachwith “human in the loop”, i.e., human approval or human annotations willbe required in cases where the annotation confidence is low, or perother business logic decision or metric. In some embodiments, the dataannotations may be generated in a manual approach, i.e., using humanannotators to generate the required annotations using convenientannotation tools. Next, at step 764, the annotated data ispre-processed, for example, by one or more of checking for and handlingnull values, imputation, standardization, handling categoricalvariables, one-hot encoding, resampling, scaling, filtering, outlierremoval and other data manipulations, to prepare the data for furtherprocessing. At optional step 765, extraction (or selection) of variousfeatures of the data may be performed, as explained hereinabove. At step766, the data and/or features extracted therefrom is divided to trainingdata (“training set”), which will be used to train the model, andtesting data (“testing set”), which will not be introduced into themodel during model training so it can be used as “hold-out” data to testthe final trained model before deployment. The training data may befurther divided into a “train set” and a “validation set”, where thetrain set is used to train the model and the validation set is used tovalidate the model's performance on unseen data, to allowoptimization/fine-tuning of the training process'configuration/hyperparameters during the training process. Examples forsuch hyperparameters may be the learning-rate, weights regularization,model architecture, optimizer selection, etc. In some embodiments, thetraining process may include the use of a Cross-Validation (CV) methodsin which the training data is divided into a “train set” and a“validation set”, however, upon training completion, the trainingprocess may repeat multiple times with different selections of “trainset” and “validation set” out of the original training data. The use ofCV may allow a better validation of the model during the trainingprocess as the model is being validated against different selections ofvalidation data. At optional step 767, data augmentation is performed.Data augmentation may include, for example, generation of additionaldata from/based on the collected or annotated data. Possibleaugmentations that may be used for image data are: rotation, flip, noiseaddition, color distribution change, crop, stretch, etc. Augmentationsmay also be generated using other types of data, for example by addingnoise or applying a variety of mathematical operations. In someembodiments, augmentation may be used to generate synthetic data samplesusing synthetic data generation approaches, such as distribution based,Monte-Carlo, Variational Autoencoder (VAE),Generative-Adversarial-Network (GAN), etc. Next, at step 768, the modelis trained, wherein the training may be performed “from scratch” (i.e.,an initial/primary model with initialized weights is trained based onall relevant data) and/or utilizing existing pre-trained models asstarting points and training them only on new data. At step 769, thegenerated model is validated. Model validation may include evaluation ofdifferent model performance metrics, such as accuracy, precision,recall, F1 score, AUC-ROC, etc., and comparison of the trained modelagainst other existing models, to allow deployment of the model whichbest fits the desired solution. The evaluation of the model at this stepis performed using the testing data (“test set”) which was not used formodel training nor for hyperparameters optimization and best representsthe real-world (unseen) data. At step 770, the trained model is deployedand integrated or utilized with the inference module to generate outputbased on newly collected data, as detailed herein.

According to some embodiments, as more data is collected, the trainingdatabase may grow in size and may be updated. The updated database maythen be used to re-train the model, thereby updating/enhancing/improvingthe model's output. In some embodiments, the new instances in thetraining database may be obtained from new clinical cases or proceduresor from previous (existing) procedures that have not been previouslyused for training. In some embodiments, an identified shift in thecollected data's distribution may serve as a trigger for the re-trainingof the model. In other embodiments, an identified shift in the deployedmodel's performance may serve as a trigger for the re-training of themodel. In some embodiments, the training database may be a centralizeddatabase (for example, a cloud-based database), or it may be a localdatabase (for example, for a specific healthcare facility). In someembodiments, learning and updating may be performed continuously orperiodically on a remote location (for example, a cloud server), whichmay be shared among various users (for example, between variousinstitutions, such as hospitals). In some embodiments, learning andupdating may be performed continuously or periodically on a single or ona cohort of medical devices, which may constitute an internal network(for example, of an institution, such as a hospital). For example, insome instances, a validated model may be executed locally on processorsof one or more medical systems operating in a defined environment (forexample, a designated institution, such as a hospital), or on localonline servers of the designated institution. In such case, the modelmay be continuously updated based on data obtained from the specificinstitution (“local data”), or periodically updated based on the localdata and/or on additional external data, obtained from other resources.In some embodiments, federated learning may be used to update a localmodel with a model that has been trained on data from multiplefacilities/tenants without requiring the local data to leave thefacility or the institution.

Reference is now made to FIGS. 8A-8B, which show an exemplary inferencemodule (FIG. 8A) and an exemplary inference process (FIG. 8B), accordingto some embodiments.

As shown in FIG. 8A, inference module 80 may include two main hardwarecomponents/units: at least one memory unit 82 and at least oneprocessing unit 84, which are functionally and/or physically associated.Inference module 80 is essentially configured to run collated data intothe trained model to calculate/process an output/prediction. Memory 82may include any type of accessible memory (volatile and/ornon-volatile), configured to receive, store and/or provide various typesof data and executable instructions, to be processed by processing unit84, which may include any type of at least one suitable processor. Insome embodiments, the memory 82 and the processing unit 84 may befunctionally or physically integrated, for example, in the form of aStatic Random Access Memory (SRAM) array. In some embodiments, thememory is a non-volatile memory having stored therein executableinstructions (for example, in the form of a code, service, executableprogram and/or a model file containing the model architecture and/orweights) that can be used to perform a variety of tasks, such as datacleaning, required pre-processing steps and inference operation (asdetailed below) on new data to obtain the model's prediction or result.As shown in FIG. 8A, memory 82 may be configured to accept/receive,store and/or provide various types of data values or parameters relatedto the data as well as executable algorithms (in the case of machinelearning based algorithms, these may be referred to as “trainedmodels”). Memory unit 82 may store or accept new acquired data 822,which may be raw (primary) data that has been collected, as detailedherein. Memory module 82 may further store metadata 824 related to theraw data. Such metadata may include a variety of parameters/valuesrelated to the raw data, such as, but not limited to: the specificdevice which was used to provide the data, the time the data wasobtained, the place the data was obtained (such as specific operationroom, specific institution, etc.), and the like. Memory 82 may furtherstore the trained model(s) 826. The trained models may be the modelsgenerated and deployed by a training module, such as training module 70of FIG. 7A. The trained model(s) may be stored, for example in the formof executable instructions and/or model file containing the model'sweights, capable of being executed by processing unit 84. Processingunit 84 of inference module 80 may include at least one processor,configured to process the new obtained data and execute a trained modelto provide corresponding results (detailed in FIG. 8B). Thus, as shownin FIG. 8A, processing unit 84 is configured at least to performpre-processing of the data 842, which may include actions for preparingthe data stored in memory 82 for downstream processing, such as, but notlimited to, checking for and handling null values, imputation,standardization, handling categorical variables, one-hot encoding,resampling, scaling, filtering, outlier removal etc. In someembodiments, processing unit 84 may further be configured to extractfeatures 844 from the acquired data, using techniques such as, but notlimited to, Principal Components Analysis (PCA), Independent ComponentAnalysis (ICA), Linear Discriminant Analysis (LDA), Locally LinearEmbedding (LLE), t-distributed Stochastic Neighbor Embedding (t-SNE),Unified Manifold Approximation and Projection (UMAP) and/orAutoencoders, etc. Feature extraction may be executed using featureengineering methods in which mathematical tools are used to extractdomain-knowledge features from the raw data, for example: statisticalfeatures such as mean, variance, ratio, frequency etc. and/or visualfeatures such as dimension or shape of certain objects in an image.Alternatively, or additionally, the processing unit 84 may be configuredto perform feature selection. Processing unit 84 may further beconfigured to execute the model on the collected data and/or featuresextracted therefrom, to obtain model results 846. In some embodiments,the processing unit 84 may further be configured to execute a businesslogic 848, which can provide further fine-tuning of the model resultsand/or utilization of the model's results to a variety of automateddecisions, guidelines or recommendations supplied to the user.

Reference is now made to FIG. 8B, which shows steps in an exemplaryinference process 86, executed by a suitable inference module (such asinference module 80 of FIG. 8A). As shown in FIG. 8B, at step 861, newdata is acquired/collected from or related to newly executed medicalprocedures. The new data may include any type of raw (primary) data, asdetailed herein. At optional step 862, suitable trained model(s)(generated, for example by a suitable training model in a correspondingtraining process) may be loaded, per task(s). This step may be requiredin instances in which computational resources are limited and only asubset of the required models or algorithms can be loaded into RAMmemory to be used for inference. In such cases, the inference processmay require an additional management step responsible to load therequired models from storage memory for a specific subset of inferencetasks/jobs, and once inference is completed, the loaded models arereplaced with other models that will be loaded to allow an additionalsubset of inference tasks/jobs. Next, at step 863, the raw datacollected in step 861 is pre-processed. In some embodiments, thepre-processing steps may be similar or identical to the pre-processingstep preformed in the training process (by the training module), tothereby allow the data to be processed similarly by the two modules(i.e., training module and inference module). In some embodiments, thisstep may include actions such as, but not limited to, checking for andhandling null values, imputation, standardization, handling categoricalvariables, one-hot encoding, etc., to prepare the input data foranalysis by the model(s). Next, at optional step 864, extraction offeatures from the data may be performed using, for example, PrincipalComponents Analysis (PCA), Independent Component Analysis (ICA), LinearDiscriminant Analysis (LDA), Locally Linear Embedding (LLE),t-distributed Stochastic Neighbor Embedding (t-SNE), Unified ManifoldApproximation and Projection (UMAP) and/or Autoencoders, etc.Alternatively, or additionally, feature selection may be executed. Atinference step 865, the results of the model are obtained, i.e., themodel is executed on the processed data to provide correspondingresults. At optional step 866, fine-tuning of the model results may beperformed, whereby post-inference business logic is executed. Executionof post-inference business logic refers to the utilization of themodel's results to a variety of automated decisions, guidelines orrecommendations supplied to the user. Post-inference business logic maybe configured to accommodate specific business and/or clinical needs ormetrics, and can vary between different scenarios or institutions basedon users' or institutions' requests or needs.

At step 867, the model results may be utilized in various means,including, for example, providing prediction, prevention and/or earlydetection of various clinical conditions (e.g., pneumothorax, breathinganomalies, bleeding, etc.), enhancing the operation of the automatedmedical device (e.g., enabling automatic target tracking and closed-loopsteering based on the tracked real-time position of the target, etc.),providing recommendations regarding various device operations (includingrecommending one or more optimal entry points, recommending optimizedtrajectories or modes of operation, etc.), and the like, as furtherdetailed hereinabove.

In some embodiments, inference operation may be performed on a singledata instance. In other embodiments, inference operation may beperformed using a batch of multiple data instances to receive multiplepredictions or results for all data instances in the batch. In someembodiments, an ensemble of models or algorithms can be used forinference, where the same input data is processed by a group ofdifferent models and results are being aggregated using averaging,majority voting or the like. In some embodiments, the model can bedesigned in a hierarchical manner where input data is processed by aprimary model and based on the prediction or result of the primarymodel's inference, the data is processed by a secondary model. In someembodiments, multiple secondary models may be used, and hierarchy mayhave more than two levels.

According to some embodiments, the methods and systems disclosed hereinutilize data-driven methods to create algorithms based on variousdatasets, including, functional, anatomical, clinical, diagnostic,demographic and/or administrative datasets. In some embodiments,artificial intelligence (e.g., machine-learning) algorithms are used tolearn the complex mapping/correlation/correspondence between themultimodal (e.g., data obtained from different modalities, such asimages, logs, sensory data, etc.) input datasets parameters (procedure,clinical, operation, patient related and/or administrative information),to optimize the clinical procedure's outcome or any other desiredfunctionalities. In some embodiments, the systems and methods disclosedherein determine such optimal mapping using various approaches, such as,for example, a statistical approach, and utilizing machine-learningalgorithms to learn the mapping/correlation/correspondence from thetraining datasets.

Reference is now made to FIGS. 9A-9C, which show exemplary medicalprocedural implications, which may be automatically analyzed/enhanced bya data-analysis algorithm, according to some embodiments. Reference ismade to FIG. 9A, which shows a pictogram of a demonstration of anindication/recommendation of “no-fly” zones 90 and 92, which are regionsto be avoided during the medical procedure (insertion of a needle in theexample shown in FIG. 9A), in order to prevent damage to avital/sensitive organ (aorta and spine, in the example shown in FIG. 9A)or to the medical instrument. Thus, based on collected primary datasetsand training sets, an algorithm generated based on data science and/ormachine learning tools (including, for example, image analysis, such asclassification, object detection and/or segmentation of scans andcorrelation of body movement during the procedure) can recommend such“no-fly” zones, to thereby enhance the safety of the medical procedure,as described in further detail hereinbelow.

Reference is made to FIG. 9B, which shows a pictogram of a demonstrationof real-time target movement during a needle insertion procedure. Duringthe insertion/steering procedure, the target 94 may move, for example,due to body motion during the breathing cycle, or as a result from theinsertion of the needle 96 into the tissue, thus it is of vitalimportance to determine the real-time location of the target 94 in orderto ensure a safe and successful procedure. Accordingly, based oncollected primary datasets and training sets, a data-analysis algorithmcan predict the real-time movement of the target 94, and the initialplanning and/or real-time updating of the trajectory can then be based,inter alia, on the target's predicted movement, thereby enhancing thesafety and accuracy of the medical procedure.

Reference is made to FIG. 9C, which shows a pictogram of a demonstrationof checkpoints 93 located along a trajectory 95 for inserting a medicalinstrument (e.g., needle) toward an internal target. Checkpoints may beused to pause the insertion of the medical instrument and initiateimaging of the region of interest, to verify the position of theinstrument, target and/or obstacle/s. The trade-off of utilizing manycheckpoints is prolonged procedure time, as well as repeated exposure toradiation. On the other hand, too little checkpoints may affect theaccuracy and safety of the medical procedure. Accordingly, based on thecollected datasets and the training data, a data-analysis algorithm canbe trained to recommend optimal checkpoint locations during the planningphase and/or during the procedure, as described in further detailhereinbelow.

In some embodiments, the algorithm may be a generic algorithm, which isagnostic to specific procedure characteristics, such as type ofprocedure, user, service provider or patient. In some embodiments, thealgorithm may be customized to a specific user (for example, preferencesof a specific healthcare provider), a specific service provider (forexample, preferences of a specific hospital), a specific population (forexample, preferences of different age groups), a specific patient (forexample, preferences of a specific patient), and the like. In someembodiments, the algorithm may be combined a generic portion and acustomized portion.

Reference is now made to FIG. 10 , which shows a block diagram 100 ofexemplary datasets and parameters used for generating a checkpoint AImodel 1002 for optimizing checkpoint locations, and an exemplary output1010 of the checkpoint model 1002, according to some embodiments. Asdetailed above, in many cases it is imperative to determine optimalcheckpoint (CP) locations (i.e., the number of checkpoints and theirpositioning along the planned and/or updated trajectory), to allowmaximal accuracy and minimal radiation exposure and/or procedure time.To this aim, one or more data-analysis algorithms, for example CP model1002, may be generated, based on various datasets and parameters. Forexample, input data may include clinical/procedure and patient-relateddata 1004, device operation data 1006 and ground truth annotations (alsoreferred to as “target variables”) 1008. The clinical/procedure data1004 may include values and/or parameters, such as, but not limited to:procedure type (e.g., biopsy, ablation, fluid drainage, etc.), targetorgan, target type, target size, instrument type (e.g., introducer,biopsy needle, ablation probe, etc.), instrument gauge, instrument tiptype (e.g., diamond tip, bevel tip), images (e.g., CT scans) andscanning parameters, respiration signal and status, respirationabnormalities, patient specific parameters (age, gender, race, BMI,clinical condition, etc.). The device operation data 1006 may includevalues and/or parameters such as, but not limited to: entry point,insertion angle (relative to one or more axis), target position, targetposition updates, instrument trajectory (planned and updated, ifupdated), trajectory path (i.e., tissue transitions along thetrajectory), actual instrument trajectory (i.e., actual instrumentposition at each CP, optionally including a time stamp), position of thedevice relative to the patient's body, steering steps timing, procedureaccuracy (e.g., instrument (e.g., the tip thereof) distance to target),target error, medical images, medical imaging parameters per scan,sensor(s) measurements, errors indicated during the steering procedure,software logs, motion control traces, automated medical deviceregistration logs, medical instrument detection logs, homing and BITresults, and the like. The data annotations 1008 may include valuesand/or parameters such as, but not limited to: procedure duration (totaland/or by insertion steps), total radiation dose in the procedure, totalradiation dose for the instrument steering phase of the procedure,average radiation dose per scan (i.e., per checkpoint), number ofcheckpoints, checkpoint positions, checkpoint updates, checkpoint errors(e.g., the deviation of the actual CP location (the location theinstrument tip actually reached) from the planned CP location, durationof the steering phase of the procedure, procedure accuracy (e.g.,instrument tip-to-target distance), complications occurrence (yes/no),complications detection time, organs segmentation masks and/or boundingboxes and/or locations, tissues segmentation mask and/or bounding boxesand/or locations, target contours and/or bounding box and/or locations,“no-fly” zones masks and/or bounding boxes and/or locations, bloodvessels mask and/or bounding boxes and/or locations, instrumentsegmentation mask and/or bounding box and/or locations, and the like.The various input datasets and the parameters derived therefrom (some orall) may be utilized to generate one or more CP models 1002. In someembodiments, each or at least some of the parameters are attributed anappropriate weight which is taken into account in generating the CPmodel 1002. The generated model can thus provide recommendations and/orassessments regarding the optimal checkpoint locations 1010A. In someembodiments, the model 1002 may provide additional assessments and/orpredictions, such as, but not limited to: the estimated duration of theprocedure 1010B (for example, the estimated time required for steeringan instrument to the target) and the estimated total radiation dose1010C (associated with CT scans, for example) during the procedureand/or during the steering phase of the procedure. In some embodiments,the recommendations may be implemented automatically orsemi-automatically in a corresponding medical procedure. In someembodiments, the recommendations may be provided to the user, e.g.,visually on a graphical user interface (GUI) on a display of the medicaldevice/system, a controller system, a mobile device, a Virtual Reality(VR) device and/or an Augmented Reality (AR) device, for his/herapproval prior to implementation. In a fully automated process, theadditional output may be for information only (if provided at all),whereas in a semi-automatic process (or manually involved process), theadditional output is provided to the physician, so that the physiciancan use this data to decide if to accept the recommendation or changethe CP locations (for example, move any of the CPs, add or delete one ormore CPs). According to some embodiments, the output 1010 may beprovided during the planning stage of the procedure, with the mainoutput being a recommendation of the optimal CP locations (i.e., numberof CPs and their positions along the planned trajectory).

Reference is now made to FIG. 11 , which shows a block diagram 110illustrating an exemplary method of generating (training) an AI modelfor optimizing checkpoint locations along an instrument trajectory in animage-guided procedure for inserting a medical instrument to an internaltarget, according to some embodiments. As described hereinabove, settingmany checkpoints along the trajectory can increase the accuracy of theprocedure (i.e., distance from the tip of the instrument to the target),since in each checkpoint real-time images (e.g., scans) may be obtained,and should there be a need (e.g., due to target movement), thetrajectory can be updated. The trade-off, however, is prolongedprocedure time, as well as repeated exposure to radiation. Therefore, insome embodiments, determining the optimal checkpoint locations shouldtake into account the predicted accuracy of the procedure, the predictedradiation dose per initiated imaging (i.e., at each checkpoint) and thepredicted duration of the steering phase of the procedure. In addition,to increase the safety of the patient, the predicted risk level of theprocedure (e.g., probability of complications) may also be taken intoaccount, according to some embodiments. To this end, in someembodiments, the training process of the checkpoint location model (alsoreferred to as “checkpoint model” or “CP model”) may include apreliminary phase of training one or more of the following individualmodels: an accuracy estimation model, a radiation dose estimation model,a duration estimation model, a risk estimation model, and anycombination thereof. The input for training each of these individualmodels may include any relevant input obtained from previous procedures,such as, but not limited to, the data described in FIG. 10 hereinabove.In some embodiments, the target variable (“ground truth) for trainingthe accuracy model is the procedure accuracy (e.g., instrumenttip-to-target accuracy). In some embodiments, the target variable fortraining the radiation dose model is the average radiation dose percheckpoint. In some embodiments, the target variable for training theduration model is the duration of the steering phase of the procedure.In some embodiments, the target variable for training the risk model isthe occurrence of complications during the procedure. It can beappreciated that for each individual model the target variable is notincluded in the input variables used for the training process of theindividual model.

According to some embodiments, in the second phase of the checkpointmodel training process the model is trained to predict CP locations assimilar as possible to the ground truth CP locations (i.e., with minimalerror from the actual CP locations along the trajectory in previousprocedures). In some embodiments, the CP model is trained to output anoptimized CP locations, i.e., not only to accurately predict the groundtruth CP locations, but to provide a CP locations recommendation thatwill also result in the maximal possible tip-to-target accuracy, minimaltotal radiation dose during the steering phase, minimal steering phaseduration and minimal risk for clinical complications during instrumentsteering. In some embodiments, such training may be executed using aloss function, e.g., a Multi-Loss scheme. In some embodiments, suchtraining may be executed using Ensemble Learning methods. In someembodiments, such training may be executed using a Multi-Outputregression/classification approach. In some embodiments, Multi-Tasklearning may be used. As shown in FIG. 11 , which illustrates trainingexecuted using a Multi-Loss scheme, input data 1102, such as the datadescribed above, is used to train the CP model 1104 to predict the CPlocations 1106 (ground truth). The predicted CP locations 1106, togetherwith the original input data 1102, is then used as input to theindividual models 1108—accuracy model, dose model, duration model andrisk model, to generate accuracy, radiation dose, duration and riskpredictions 1110, respectively. The individual models' predictions 1110,together with the CP model's prediction, are then used to calculate aloss function 1112, aimed to minimize the CP locations prediction error,maximize the tip-to-target accuracy, minimize the radiation dose,minimize the duration and minimize the risk. The generated weighted lossrepresents the model's prediction error, which may be used to fine-tuneor adjust the CP model's 1104 weights as part of the training process.

In some embodiments, only one or more of the individual models describedabove are used in the training process of the CP model. For example, insome embodiments only the accuracy and duration models may be used,whereas in other embodiments only the accuracy and dose models may beused. Further, the weights (coefficients) used in the Multi-Lossfunction 1112 may be adjusted according to certain needs and/orpreferences. For example, if minimal radiation dose and/or minimalduration have a higher priority than CP locations prediction accuracy,tip-to-target accuracy and/or risk, the dose and duration may be givenhigher coefficients during the training process, such that they willhave a greater impact on the CP locations recommendations. In someembodiments, different CP models may be trained for different needsand/or preferences. For example, one CP model may be trained to generatea CP locations recommendation that will allow the highest achievabletip-to-target accuracy, another CP model may be trained to generate a CPlocations recommendation that will allow the lowest achievable radiationdose, a further CP model may be trained to generate a CP locationsrecommendation that will result in the shortest achievable duration,etc. In some embodiments, a single CP model may be trained and deployed,and the coefficients used in the Multi-Loss function 1112 may beadjusted during inference, i.e., during use of the CP model to generatea CP locations recommendation for a specific procedure. Theneed/preference upon which the coefficients may be fine-tuned may beassociated with, for example, a specific procedure type (e.g., biopsy,fluid drainage, etc.), a specific target type, a specific user, aspecific population, a specific user, etc.

Reference is now made to FIG. 12 , which shows a flowchart 120illustrating the steps of a method of utilizing a checkpoint model (an“inference” process) for optimizing checkpoint locations along atrajectory, according to some embodiments. At step 1202, a plannedtrajectory from an entry point to a target is obtained. At step 1204,boundaries between tissue layers along the trajectory are detected. Insome embodiments, at optional step 1206, sections along the plannedtrajectory through which the instrument should be steered in “one shot”(for example, crossing the lung's pleura), thus no checkpoints are to bepositioned along such sections, are defined. At step 1208, the scanvolume and the radiation dose per checkpoint are, optionally, estimated.The scan volume may be estimated based, for example, on the position ofthe automated device relative to the subject's body (specifically, theposition of the device's registration elements relative to the target),the insertion angle, the type and size of the target, etc. The radiationdose per checkpoint may be estimated based, for example, on theestimated scan volume and the planned imaging device configuration(e.g., intensity, slice thickness, resolution, etc.). In someembodiments, the scan volume may be estimated using an algorithm/modelthat was trained using data from previous procedures. Next, at step1210, data and parameters obtained and/or calculated in the previoussteps are used as input for the deployed CP model and the model'sresults are obtained. It can be appreciated that additional data may beused as input for the CP model, as described in detail hereinabove. Atstep 1212, checkpoints are set along the planned trajectory based on theresults of the CP model. In some embodiments, the user may be promptedto confirm the set locations of the checkpoints. In some embodiments,the use may be able to adjust the set locations (as recommended by theprocessor). At step 1214, if the images (e.g., CT scans) obtained uponthe instrument reaching a certain checkpoint during the steeringprocedure show that the target has moved from its initial position (orfrom its previous position as identified in images obtained at aprevious checkpoint) and/or if the trajectory is updated due targetmovement, due to deviation of the instrument from the planned trajectoryabove a predetermined threshold and/or due to an obstacle identifiedalong the planned trajectory, an updated recommendation for thelocations of the subsequent checkpoints may be obtained from the CPmodel, and at step 1216, the locations of the subsequent checkpoints maybe adjusted according to the updated results, if necessary (e.g., one ormore checkpoints may be added or removed, the distance between two ormore checkpoints and/or between the last checkpoint and the target maybe adjusted, etc.). In some embodiments, the user may be prompted toconfirm the adjusted locations of the subsequent checkpoints. In someembodiments, the use may be able to further adjust the adjustedlocations of the subsequent checkpoints (as recommended by theprocessor).

Reference is now made to FIG. 13 , which shows a flowchart 130illustrating the steps of a method of utilizing (“inference” process) anAI model for creating a “no-fly” zone map, according to someembodiments. In some embodiments, generating the AI model for creating a“no-fly” zone map (also referred to as “no-fly” zone model”) may includetraining the model to predict “no-fly” zones as similar as possible tothe ground truth “no-fly” zones map (i.e., with minimal error from theactual “no-fly” zones annotation map in previous similar procedures oradditional relevant collected data available for training). In someembodiments, generating the “no-fly” zone model may include apreliminary phase, in which one or more individual models are trained.Such individual models may include an instrument's tip-to-targetaccuracy estimation model, a steering duration estimation model and/or arisk estimation model. In some embodiments, the target variable(“ground-truth) for training the tip-to-target accuracy model may be theprocedure accuracy (e.g., instrument tip-to-target accuracy). In someembodiments, the target variable for training the risk model may be theoccurrence of complications during the procedure. In some embodiments,the target variable for training the steering duration model may be theduration of the steering phase of the procedure. In some embodiments,the target variable for training the steering duration model may be thesteering duration given a certain trajectory. In some embodiments, thetrajectory may be estimated, at least in part, based on the “no-fly”zones predictions (recommendations). For example, a first “no-fly” zonesprediction may enable a linear trajectory, whereas a second “no-fly”zones prediction may require a non-linear trajectory. As a lineartrajectory is always the shortest route from the entry point to thetarget (given the same entry point and target positions), the first“no-fly” zones prediction also results in a shorter steering durationthan the steering duration resulting from the second “no-fly” zonesprediction. In embodiments in which generating the “no-fly” zone modelincludes training one or more individual models, the second phase oftraining the “no-fly” zone model may be executed using a loss function,e.g., Multi-Loss scheme, Ensemble Learning methods, Multi-Outputregression/classification approach, Multi-Task Learning and the like. Insome embodiments, the “no-fly” zone model may be trained using aMulti-Loss scheme, such that the “no-fly” zone map predicted by the“no-fly” zone model, together with the original input data, may be usedas input to the individual models. The individual models' predictions,together with the “no-fly” zone model's prediction, may then be used tocalculate a loss function, aimed to minimize the “no-fly” zonesprediction error while, for example, minimizing the steering duration,maximizing the expected tip-to-target accuracy and minimizing the risk.The generated weighted loss represents the model's prediction error,which may be used to fine-tune or adjust the “no-fly” zones model'sweights as part of the training process.

In some embodiments, only one or more of the individual models describedabove are used in the training process of the “no-fly” zone model. Forexample, in some embodiments only the accuracy and duration models maybe used, whereas in other embodiments only the accuracy and risk modelsmay be used. Further, the weights (coefficients) used in the lossfunction may be adjusted according to certain needs and/or preferences.For example, if minimal risk has a higher priority than “no-fly” zonesprediction accuracy, tip-to-target accuracy and/or steering duration,risk may be given a higher coefficient during the training process, suchthat it will have a greater impact on the “no-fly” zones recommendation.In some embodiments, different “no-fly” zones models may be trained fordifferent needs and/or preferences. For example, one “no-fly” zonesmodel may be trained to generate a “no-fly” zones recommendation thatwill allow the highest achievable tip-to-target accuracy, another“no-fly” zones model may be trained to generate a “no-fly” zonesrecommendation that will allow the lowest achievable risk to thepatient, a further “no-fly” zones model may be trained to generate a“no-fly” zones recommendation that will result in the shortestachievable duration, etc. In some embodiments, a single “no-fly” zonesmodel may be trained and deployed, and the coefficients used in theMulti-Loss function may be adjusted during inference, i.e., during useof the “no-fly” zones model to generate a “no-fly” zones recommendationfor a specific procedure. The need/preference upon which thecoefficients may be fine-tuned may be associated with, for example, aspecific procedure type (e.g., biopsy, fluid drainage, etc.), a specifictarget type, a specific user, specific patient characteristics, etc.

As shown in FIG. 13 , at step 1302, images of a region of interest areobtained from an imaging system, such as a CT scanner, ultrasound, Mill,CBCT, etc. At step 1304, a segmentation map is calculated. Thecalculation may be done using a ML/DL based segmentation model capableof generating pixel-based 2D or 3D segmentation. In some embodiments, asemantic segmentation model may be used. In some embodiments, instancesegmentation may be used. In some embodiments, the different segmentsand/or objects in the image(s) are classified to classes, such asorgans, blood vessels, lesions, etc. In some embodiments, theclassification may be pixel/voxel based. At step 1306, “risky” segments(also referred to as “sensitive segments” or “obstacles”) areidentified. Such segments may include, for example, bones, bloodvessels, specific tissues, specific organs, etc. At optional step 1308,the movement range of the “risky” segments due to respiration motion maybe estimated. The estimation may be based solely on image processing orit may be calculated using a separate data-analysis model. In someembodiments, the planning stage of the medical procedure (e.g., animage-guided interventional procedure) may include estimating theexpected movement of the patient due to breathing based on a sequence ofpre-operative images, and planning the trajectory for the instrumentaccordingly, as disclosed, for example, in co-owned U.S. Pat. No.10,245,110, which is incorporated herein by reference in its entirety.Next, at step 1310, data and parameters obtained and/or calculated inthe previous steps are used as input for the “no-fly” zone model and themodel's results are obtained. It can be appreciated that additional datamay be used as input for the model, as described in detail hereinabove.At step 1312, a “no-fly” zone map is created based on the results of the“no-fly” zone model. At step 1314, if the images obtained from theimaging system during the steering procedure show that the target hasmoved from its initial position (or from its previously identifiedposition) and/or if the trajectory is updated due target movement, dueto deviation of the instrument from the planned trajectory above apredetermined threshold and/or due to an obstacle identified along theplanned trajectory, updated results may be obtained from the “no-fly”zone model, and at step 1316, the “no-fly” zone map may be adjustedaccording to the updated model results, if necessary. In someembodiments, if checkpoints are set along the trajectory such that uponthe instrument reaching a checkpoint, advancement of the instrument ispaused and imaging is initiated, steps 1312 and 1314 may be executed ateach checkpoint. In some embodiments, if the medical procedure isperformed under continuous or substantially continuous imaging (e.g.,using a CT fluoroscopy system or an ultrasound system), steps 1312 and1314 may be executed continuously or at defined temporal or spatialintervals during the procedure.

Reference is now made to FIG. 14 , which shows a block diagram 140 ofdatasets and parameters used for generating an AI model for predictionand/or detection of pneumothorax (also referred to as “pneumothoraxprediction model”, “pneumothorax detection model” or “pneumothoraxmodel”) 1402, according to some embodiments. A pneumothorax occurs whenair enters the pleural sac, i.e., the space between the lung and thechest wall, pushing on the outside of the lung and causing the lung tocollapse. Pneumothorax can be a complete lung collapse or a partial lungcollapse, and it can inadvertently occur during medical procedures thatinvolve the insertion of a medical instrument (e.g., needle) into thechest, such as lung biopsy. Pneumothorax may be life-threatening, thusit may be advantageous to train AI model(s) to predict and/or detect theoccurrence of pneumothorax during a medical procedure and, optionally,recommend actions that may prevent the occurrence of pneumothorax,prevent worsening of a developing pneumothorax and/or enable earlytreatment to an existing pneumothorax. Such AI model(s) may be employed,for example, when a medical instrument is inserted into the lung for thepurpose of performing a lung biopsy or in a medical procedure which isadjacent to the pleura.

In some embodiments, the input datasets may include, for example, butnot limited to: data related to clinical procedure and patient relateddata 1404, such as, target (e.g., lesion) size, target depth, medicalinstrument (needle) type and gauge, needle tip type (e.g., diamond,bevel), respiration signals, respiration abnormalities, patientcharacteristics (age, gender, race, lung function, BMI, previous lungprocedures, clinical condition, smoking habits, etc.); data related tothe medical device and its operation 1406, including, for example,motors' current traces (i.e. logs of motors' performance data),procedure timing, skin to target time, entry and target positions,trajectory length, target movements and paths updates, number andposition of checkpoints, errors and correction of checkpoints, images(e.g., CT scans) generated during the procedure (e.g., at checkpoints),magnitude of lateral steering of the medical instrument, medical deviceposition, insertion angles, final tip-to-target accuracy (distance,depth, lateral), fissure crossed, bulla crossed, pleura crossed,distance of target from lung wall, patient's position (e.g., supine,prone, decubitus), location of target (e.g., in the right lung or theleft lung), etc. In addition, data annotations 1408 are further utilizedfor model training and validation, including, for example, whether apneumothorax has been detected in past (similar) procedures,pneumothorax size, pneumothorax location (e.g., as marked on thescan/s), etc. Once the pneumothorax model is generated and validated,based on the various datasets, output (results/predictions) 1410 may beprovided. Such output may be, for example, the probability ofpneumothorax 1410A, the estimated pneumothorax size 1410B, potentialmodifications 1410C which could reduce the probability of pneumothorax,and the like, or any combination thereof.

In some embodiments, the output of the model 1402 may be communicated toa user, for example, visually on a graphical user interface (GUI) on adisplay of the medical device/system, a controller system, a mobiledevice, a Virtual Reality (VR) device and/or an Augmented Reality (AR)device, and the like. In some embodiments, the output (for example, arecommendation) of the model 1402 may be communicated to a healthcareprovider, which may allow (or not allow) the execution of therecommendation. In some embodiments, the execution of the recommendationissued by the model 1402 may be performed automatically after beingcommunicated to an automated medical device.

Reference is now made to FIG. 15 , which shows a block diagram 150illustrating an exemplary method of generating (training) an AI modelfor prediction and/or detection of pneumothorax (“pneumothorax model”).As shown in FIG. 15 , input data 1502, such as input described in FIG.14 , is used to train the pneumothorax model 1504 to estimate theprobability of pneumothorax occurrence 1506. The input data 1502 mayinclude multi-modal data collected from past procedures and arranged,where possible/applicable, as time-series together with the patient'sparameters and medical history. The time-series structure may allow theanalysis of time-dependency events in past procedures' data to betterpredict the probability for pneumothorax occurrence during a procedureand better study the impact of the different risk factors and theircorrelation to the procedure timeline. In some embodiments, specializedfeature extraction models 1504 may be used to generate meaningfuldomain-knowledge features that may, in turn, be input to the primarypneumothorax model 1506 during the training process. Such specializedfeatures extraction models 1504 may be, for example, a pleural cavityvolume (and/or size and/or shape) estimation model, a fissure and bullacrossing model, patient position model, respiration anomalies model,etc. the specialized feature extraction models 1504 may be trained onrelevant portions of the input data and their output may be input to theprimary pneumothorax model 1506 together with the remaining multi-modaldata. In some embodiments, the output of the pneumothorax model 1506 maybe prediction 1508 of the probability of pneumothorax occurrence in thecurrent procedure. This prediction, together with ground-truthannotations regarding the occurrence of pneumothorax during a procedure,may be used to calculate a loss function 1510 representing the errorbetween the pneumothorax model's prediction and the ground-truth data.During the training process, optimization of this loss function willallow the adjustment of the model's weights. In some embodiments, thepneumothorax model may be trained in a multi-task and/or multi-outputapproach. In such embodiments, the model may predict, for example, thepoint in time representing the beginning of an active pneumothoraxcondition, in addition to the probability of pneumothorax occurrence. Insome embodiments, the pneumothorax model 1506 may be trained to predictthe exact risk of pneumothorax at each point in time during theprocedures. This may require corresponding time-based annotations ofpneumothorax risk level at desired points in time throughout theprocedures in the dataset. In some embodiments, the pneumothorax modelmay be trained to predict the primary identified risk factors and/ortheir contribution to the overall pneumothorax occurrence probability.

Reference is now made to FIG. 16 , which shows a flowchart 160illustrating the steps of a method of utilizing (an “inference” process)a pneumothorax model for prediction, early detection and/or preventionof pneumothorax. At step 1602, patient data may be, optionally,obtained. Such data may include, for example, but not limited to: age,gender, BMI, smoking habits, etc. Patient data may further include thepatient's medical history, such as the patient's lung function, previousmedical procedures (specifically, lung procedures), previous occurrencesof pneumothorax, medical condition, etc. At step 1604, characteristicsof the medical instrument to be used in the procedure are obtained. Suchcharacteristics may include, for example, instrument type (e.g.,introducer, biopsy needle, ablation probe, etc.), instrument gauge,instrument tip type (e.g., diamond tip, bevel tip), etc. At step 1606,the patient's position (pose) on the procedure bed is obtained. Thepatient's pose may be, for example, supine, prone, decubitus, etc. Atstep 1608, one or more images of a region of interest are obtained froman imaging system (e.g., CT, ultrasound, MRI, X-Ray, CBCT). At step1610, a segmentation map may be calculated, according to someembodiments. The calculation may be done using a ML/DL basedsegmentation model capable of generating pixel-based 2D or 3Dsegmentation. In some embodiments, a semantic segmentation model may beused. In some embodiments, instance segmentation may be used. In someembodiments, the different segments and/or objects in the image(s) areclassified to classes, such as organs, blood vessels, lesions, etc. Insome embodiments, the classification may be pixel/voxel based. At step1612, the target, entry point and, optionally, “no-fly” zones areobtained or identified, and a trajectory for the medical instrument fromthe entry to the target, which avoid entrance into the “no-fly” zones(if marked), is calculated. In some embodiments, at least one of thetarget, entry point and “no-fly” zones may be marked on the image(s)manually by the user. In some embodiments, at least one of the target,entry point and “no-fly” zones may be identified by a processor usingimage processing and/or using dedicated data-analysis algorithms. Forexample, a “no-fly” zone map may be created using the “no-fly” zonemodel described in FIG. 13 hereinabove. In some embodiments, thetrajectory may be calculated based solely on the pre-operative images ofthe region of interest, for example as disclosed in abovementionedco-owned International Patent Application No. PCT/IL2020/051219. In someembodiments, the trajectory may be calculated using a dedicateddata-analysis algorithm, such as an AI model, using data from previous(similar) procedures. In some embodiments, the planned trajectory is aplanner trajectory (2D). In some embodiments, the planned trajectory isthree-dimensional. In some embodiments, two or more planner trajectoriesare first planned on two or more planes disposed at an angle relative toeach other, and the two or more planner trajectories are thensuperpositioned to form a planned 3D trajectory. At step 1614, thelocations of critical tissues, such as the lung, pleura, fissures,bulla(e) (if exists(s)), etc. At step 1616, the pleural cavity (sac) isdetected and its volume is determined. Once determined, the pleuralcavity volume is monitored to detect changes in the volume,specifically—enlargement thereof. The pleural cavity volume may bedetermined/monitored using image processing, sensor data and/or tissuecompliance, for example. At step 1618, the patient's respirationpatterns may, optionally, be monitored. Certain changes in the patient'srespiration patterns may be indicative of a pneumothorax developing. Atstep 1620, data and parameters obtained and/or calculated in theprevious steps are used as input for the pneumothorax model and themodel's output is obtained. It can be appreciated that additional datamay be used as input for the model, as described in detail hereinabove.In some embodiments, the model's output may include, for example, theprobability of pneumothorax and the pneumothorax size, etc. At step1622, it is determined if the probability of pneumothorax occurrence isabove a defined threshold. In some embodiments, the threshold isdetermined automatically, e.g., based, at least in part, on past similarcases (e.g., similar procedures and/or similar patient characteristics,etc.). In such embodiments, the determination if the probability ofpneumothorax is above a threshold may be included in the results of thepneumothorax model. In some embodiments, the threshold is determined bythe healthcare provider (e.g., physician), and the determination if thepneumothorax probability is above a threshold is a clinical decision ofthe healthcare provider. At step 1624, if it is determined (either bythe processor or by the healthcare provider) that the probability ofpneumothorax occurrence is above a defined threshold, then if thecalculations were executed during the planning stage of the procedure,the processor may alert the user (for example, by displaying a visualalert on the GUI and/or generating an auditory notification) and suggestmitigating actions to reduce the probability of pneumothorax occurringduring the procedure, such as repositioning the automated medicaldevice, selecting a different entry point, using a different medicalinstrument (e.g., an instrument with a higher gauge (thinner tool)),etc. In embodiments in which the probability threshold and theprobability of pneumothorax being above a threshold are determined bythe processor and are part of the output of the pneumothorax model, therecommendation of mitigating actions to reduce the probability ofpneumothorax may also be part of the output of the pneumothorax model.If mitigating actions to reduce the risk of pneumothorax cannot beexecuted, or if there are no (or insufficient) possible mitigatingactions, the processor may recommend to the user not to perform theprocedure. If mitigating actions have been implemented then, at step1626, the probability of pneumothorax is recalculated. If theprobability is now below the defined threshold, or if the initialcalculated probability was below the defined threshold, then theinstrument steering procedure is executed, and recalculation of theprobability of pneumothorax is repeated during the procedure. In someembodiments, if checkpoints have been set along the trajectory, theprobability of pneumothorax may be recalculated upon the instrumentreaching each of checkpoints. In some embodiments, the probability ofpneumothorax may be recalculated at a checkpoint only if the targetposition and/or the trajectory are updated. In some embodiments, theprobability of pneumothorax may be recalculated only upon the instrumentreaching the checkpoint closest to the lung (specifically, to thepleura). In some embodiments, if the instrument steering procedure isperformed under continuous or substantially continuous imaging (e.g.,using a CT fluoroscopy system, CBCT system or an ultrasound system), theprobability of pneumothorax may be recalculated continuously or atdefined temporal or spatial intervals during the procedure until theinstrument reaches the target.

In some embodiments, if the pneumothorax probability calculations wereexecuted during the instrument steering procedure and none (or aninsufficient number) of the risk factors can be adjusted in order toreduce the probability of pneumothorax, then if it is determined thatthe probability of pneumothorax is above the threshold, an alert may begenerated (for example, a visual alert displayed on the GUI and/or anauditory notification). In some embodiments, the processor may furtherprompt the user to stop the steering procedure. In some embodiments, theprocessor may automatically stop the steering procedure.

Reference is now made to FIG. 17 , which shows a flowchart 170illustrating the steps of a method utilizing (an “inference” process) anAI model for prediction and/or detection of internal bleeding (alsoreferred to as “bleeding model”, “internal bleeding model” or “bleedingprediction model”), according to some embodiments. Training of thebleeding prediction model may be performed similarly to the trainingdescribed in FIG. 15 hereinabove, such that multi-modal data, structuredas time-series (where applicable), is used as input to the bleedingprediction model. The model's output may be a prediction of theprobability of bleeding, and the ground-truth data regarding theoccurrence of bleeding during past procedures included in the dataset,may be used to calculate a loss function that will represent the errorbetween the model's prediction of internal bleeding and the ground-truthlabels. During the training process, optimization of the loss functionwill allow the adjustment of the model's weights for optimal prediction.In some embodiments, the bleeding model may be trained in a multi-taskand/or multi-output approach, such that it may predict, for example, thepoint in time representing the beginning of an active bleedingcondition, in addition to the probability for bleeding occurrence. Insome embodiments, the internal bleeding model may be trained to predictthe exact risk of bleeding at each point in time during the procedures.This may require corresponding time-based annotations of bleeding risklevel at desired points in time throughout the procedures in thedataset. In some embodiments, the bleeding model may be trained topredict the primary identified risk factors and/or their contribution tothe overall bleeding occurrence probability.

As shown in FIG. 17 , at step 1702, patient data may optionally beobtained. Such data may include, for example, but not limited to: age,gender, BMI, etc. Patient data may further include the patient's medicalhistory, such as the patient's medical condition, existing vasculardisease(s), previous medical procedures, previous occurrence(s) ofbleeding during medical procedures, etc. At step 1704, one or moreimages of a region of interest are obtained from an imaging system(e.g., CT, ultrasound, Mill, X-Ray, CBCT). At step 1706, a segmentationmap may be calculated, according to some embodiments. The calculationmay be done using a ML/DL based segmentation model capable of generatingpixel-based 2D or 3D segmentation. In some embodiments, a semanticsegmentation model may be used. In some embodiments, instancesegmentation may be used. In some embodiments, the different segmentsand/or objects in the image(s) are classified to classes, such asorgans, blood vessels, lesions, etc. In some embodiments, theclassification may be pixel/voxel based. At step 1708, the target, entrypoint and, optionally, “no-fly” zones are obtained or identified, and atrajectory for the medical instrument from the entry to the target,which avoid entrance into the “no-fly” zones (if marked), is calculated.In some embodiments, at least one of the target, entry point and“no-fly” zones may be marked on the image(s) manually by the user. Insome embodiments, at least one of the target, entry point and “no-fly”zones may be identified by a processor using image processing and/orusing dedicated data-analysis algorithms. For example, a “no-fly” zonemap may be created using the “no-fly” zone model described in FIG. 13hereinabove. In some embodiments, the trajectory may be calculated basedsolely on the pre-operative images of the region of interest, forexample as disclosed in abovementioned co-owned International PatentApplication No. PCT/IL2020/051219. In some embodiments, the trajectorymay be calculated using a dedicated data-analysis algorithm, such as anAI model, using data from previous (similar) procedures. In someembodiments, the planned trajectory is a planner trajectory (2D). Insome embodiments, the planned trajectory is three-dimensional. In someembodiments, two or more planner trajectories are first planned on twoor more planes disposed at an angle relative to each other, and the twoor more planner trajectories are then superpositioned to form a planned3D trajectory. At step 1710, blood vessels along the planned trajectorymay be detected. In some embodiments, the identified blood vessels arefurther classified to blood vessel types, such as artery, vein, etc. Insome embodiments, critical organs, i.e., organs which are moresusceptible to bleed, if punctured, and/or organs which, if punctured,the resultant bleeding may lead to a life-threating condition, are alsodetected and/or classified. In some embodiments, if a “no-fly” zone mapis created, the step of detecting and/or classifying blood vesselsand/or critical organs, may be part of the creation of the “no-fly” zonemap (step 1708). At step 1712, data and parameters obtained and/orcalculated in the previous steps are used as input for the bleedingmodel and the model's output is obtained. It can be appreciated thatadditional data may be used as input for the model, as described indetail hereinabove. In some embodiments, the model's output may include,for example, the probability that internal bleeding will occur duringthe medical procedure. During the planning phase of the procedure, thecalculation of the probability that internal bleeding will occur duringthe procedure may be based, for example, on the planned trajectory, thelocation of blood vessels and/or critical organs along the trajectoryand/or the patient's characteristics detailed above. At step 1714, it isdetermined if the probability that internal bleeding will occur duringthe procedure is above a defined threshold. In some embodiments, thethreshold is determined automatically, e.g., based, at least in part, onpast similar cases (e.g., similar procedures and/or similar patientcharacteristics, etc.). In such embodiments, the determination if theprobability is above a threshold may be included in the results of thebleeding prediction model. In some embodiments, the threshold isdetermined by the healthcare provider (e.g., physician), and thedetermination if the bleeding probability is above a threshold is aclinical decision of the healthcare provider. At step 1716, if it isdetermined (either by the processor or by the healthcare provider) thatthe probability of internal bleeding occurrence is above a definedthreshold, then the processor may alert the user (for example, bydisplaying a visual alert on the GUI and/or generating an auditorynotification) and suggest mitigating actions to reduce the probabilityof internal bleeding occurring during the procedure, such asrepositioning the medical device, selecting a different entry point,adjusting the “no-fly” zones, adjusting the checkpoint locations alongthe trajectory and/or recalculating the trajectory, etc. In embodimentsin which the probability threshold and the probability of bleedingoccurrence being above a threshold are determined by the processor andare part of the output of the bleeding model, the recommendation ofmitigating actions to reduce the probability of internal bleeding mayalso be part of the output of the bleeding model. After mitigatingactions have been implemented, the probability of bleeding occurringduring the procedure may be recalculated (at step 1712). If theprobability is now below the defined threshold (at step 1714), or if theinitial calculated probability was below the defined threshold, then themedical procedure is executed and, at step 1718, the probability ofbleeding occurrence is repeated during the procedure, using the internalbleeding model. In some embodiments, the output of the model during theinsertion procedure may include, instead or in addition to theprediction of bleeding occurring during subsequent steps of theprocedure, a prediction/detection that bleeding is occurring (presenttense), as well as the suspected location of the bleeding in thepatient's body and additional characteristics of the bleeding. Suchcharacteristics may be, for example, estimated bleeding rate, estimatedbleeding volume and additional characteristics which may be indicativeof the severity of the bleeding. In some embodiments, if checkpointshave been set along the trajectory, the bleeding probability may berecalculated upon the instrument reaching each of checkpoints. In someembodiments, the bleeding probability may be recalculated at acheckpoint only if there are changes in certain parameters, for example,if the target position and/or the trajectory are updated, if thecheckpoint location are adjusted, if the scan volume is changed, etc. Insome embodiments, if the instrument steering procedure is performedunder continuous or substantially continuous imaging (e.g., using a CTfluoroscopy system, CBCT system or an ultrasound system), theprobability of internal bleeding may be recalculated continuously or atdefined temporal or spatial intervals during the procedure until theinstrument reaches the target. At step 1720, it is determined if theprobability that there is (present tense) internal bleeding and/or thatbleeding will occur during following steps of the procedure is above adefined threshold, similarly to step 1714. At step 1722, if it isdetermined (either by the processor or by the healthcare provider) thatthe probability of internal bleeding occurrence is above the definedthreshold, then the processor may alert the user and present to the userthe suspected location of the bleeding (existing or predicted). In someembodiments, additional characteristics of the bleeding may be presentedto the user, such as estimated bleeding rate, etc. At step 1724, if itis decided to continue the steering procedure, either following anassessment by the processor or a clinical decision of the physician,then the probability of bleeding may be recalculated (at step 1718)continuously or at one or more checkpoints, for example, until theinstrument reaches the target. If it is decided to terminate theprocedure due to the bleeding (existing or predicted), either followingan assessment by the processor or a clinical decision of the physician,then the process ends, at step 1726.

Implementations of the systems, devices and methods described above mayfurther include any of the features described in the present disclosure,including any of the features described hereinabove in relation to othersystem, device and method implementations.

According to some embodiments, there is provided computer-readablestorage medium having stored therein data-analysis algorithm(s),executable by one or more processors, for generating one or more modelsfor providing recommendations, operating instructions and/or functionalenhancements related to operation of automated medical devices.

The embodiments described in the present disclosure may be implementedin digital electronic circuitry, or in computer software, firmware orhardware, or in combinations thereof. The disclosed embodiments may beimplemented as one or more computer programs, i.e., one or more modulesof computer program instructions, encoded on computer storage medium forexecution by, or to control the operation of, one or more dataprocessing apparatus. Alternatively or in addition, the computer programinstructions may be encoded on an artificially generated propagatedsignal, for example, a machine-generated electrical, optical orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of any one or more of the above. Furthermore, while acomputer storage medium is not a propagated signal, a computer storagemedium can be a source or destination of computer program instructionsencoded in an artificially generated propagated signal. The computerstorage medium can also be, or be included in, one or more separatephysical components or media (for example, multiple CDs, disks, or otherstorage devices).

The operations described in the present disclosure can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” as used herein may encompass alltypes of apparatus, devices, and machines for processing data, includingby way of example a programmable processor, a computer, a system on achip/s, or combinations thereof. The data processing apparatus caninclude special purpose logic circuitry, for example, an FPGA (fieldprogrammable gate array) or an ASIC (application specific integratedcircuit). The apparatus can also include, in addition to hardware, codethat creates an execution environment for the computer program inquestion, for example, code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, across-platform runtime environment, a virtual machine, or combinationsthereof. The apparatus and execution environment can realize variousdifferent computing model infrastructures, such as web services,distributed computing and grid computing infrastructures.

A computer program (also referred to as a program, software, softwareapplication, script or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astandalone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Acomputer program can be stored in a portion of a file that holds otherprograms or data, in a single file dedicated to the program in question,or in multiple coordinated files (for example, files that store one ormore modules, sub programs or portions of code). A computer program canbe deployed to be executed on one computer or on multiple computers thatare located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described herein can be performed by oneor more programmable processors, executing one or more computer programsto perform actions by operating on input data and generating output. Theprocesses and logic flows can also be performed by, and an apparatus canalso be implemented as, special purpose logic circuitry, for example, anFPGA or an ASIC. Processors suitable for the execution of a computerprogram include both general and special purpose microprocessors, andany one or more processors of any type of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. A computermay, optionally, also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, for example, magnetic, magneto optical discs, or opticaldiscs. Moreover, a computer can be embedded in another device, forexample, a mobile phone, a tablet, a personal digital assistant (PDA, agame console, a Global Positioning System (GPS) receiver, or a portablestorage device (for example, a USB flash drive). Devices suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including semiconductormemory devices, for example, EPROM, EEPROM, random access memories(RAMs), including SRAM, DRAM, embedded DRAM (eDRAM) and Hybrid MemoryCube (HMC), and flash memory devices; magnetic discs, for example,internal hard discs or removable discs; magneto optical discs; read-onlymemories (ROMs), including CD-ROM and DVD-ROM discs; solid state drives(SSDs); and cloud-based storage. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

The processes and logic flows described herein may be performed in wholeor in part in a cloud computing environment. For example, some or all ofa given disclosed process may be executed by a secure cloud-based systemcomprised of co-located and/or geographically distributed serversystems. The term “cloud computing” is generally used to describe acomputing model which enables on-demand access to a shared pool ofcomputing resources, such as computer networks, servers, softwareapplications, and services, and which allows for rapid provisioning andrelease of resources with minimal management effort or service providerinteraction.

Unless specifically stated otherwise, as apparent from the disclosure,it is appreciated that, according to some embodiments, terms such as“processing”, “computing”, “calculating”, “determining”, “estimating”,“assessing” or the like, may refer to the action and/or processes of acomputer or computing system, or similar electronic computing device,that manipulate and/or transform data, represented as physical (e.g.electronic) quantities within the computing system's registers and/ormemories, into other data similarly represented as physical quantitieswithin the computing system's memories, registers or other suchinformation storage, transmission or display devices.

It is to be understood that although some examples used throughout thisdisclosure relate to procedures for insertion of a needle into asubject's body, this is done for simplicity reasons alone, and the scopeof this disclosure is not meant to be limited to insertion of a needleinto the subject's body, but is understood to include insertion of anymedical tool/instrument into the subject's body for diagnostic and/ortherapeutic purposes, including a port, probe (e.g., an ablation probe),introducer, catheter (e.g., drainage needle catheter), cannula, surgicaltool, fluid delivery tool, or any other such insertable tool.

In some embodiments, the term medical instrument and medical tool may beused interchangeably.

In some embodiments, the term “model”, “algorithm”, “data-analysisalgorithm” and “data-based algorithm” may be used interchangeably.

In some embodiments, the terms “user”, “doctor”, “physician”,“clinician”, “technician”, “medical personnel” and “medical staff” areused interchangeably throughout this disclosure and may refer to anyperson taking part in the performed medical procedure.

It can be appreciated that the terms “subject” and “patient” may be usedinterchangeably, and they may refer either to a human subject or to ananimal subject.

In the description and claims of the application, the words “include”and “have”, and forms thereof, are not limited to members in a list withwhich the words may be associated.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this disclosure pertains. In case of conflict, thepatent specification, including definitions, governs. As used herein,the indefinite articles “a” and “an” mean “at least one” or “one ormore” unless the context clearly dictates otherwise.

It is appreciated that certain features of the disclosure, which are,for clarity, described in the context of separate embodiments, may alsobe provided in combination in a single embodiment. Conversely, variousfeatures of the disclosure, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable sub-combination or as suitable in any other describedembodiment of the disclosure. No feature described in the context of anembodiment is to be considered an essential feature of that embodiment,unless explicitly specified as such.

Although steps of methods according to some embodiments may be describedin a specific sequence, methods of the disclosure may include some orall of the described steps carried out in a different order. The methodsof the disclosure may include a few of the steps described or all of thesteps described. No particular step in a disclosed method is to beconsidered an essential step of that method, unless explicitly specifiedas such.

The phraseology and terminology employed herein are for descriptivepurpose and should not be regarded as limiting. Citation oridentification of any reference in this application shall not beconstrued as an admission that such reference is available as prior artto the disclosure. Section headings are used herein to easeunderstanding of the specification and should not be construed asnecessarily limiting.

1-40. (canceled)
 41. A computer-implemented method of generating acheckpoint locations model for optimizing locations of checkpoints alonga trajectory in an image-guided procedure for inserting a medicalinstrument toward a target in a body of a patient, the methodcomprising: collecting one or more datasets, at least one of the one ormore datasets being related to an automated medical device configured tosteer a medical instrument toward a target in the body of a patientand/or to operation thereof; creating a training set comprising at leasta portion of the one or more datasets and one or more target parametersrelating to checkpoint locations along a trajectory in one or moreprevious image-guided procedures for inserting a medical instrumenttoward a target in a body of a patient; training the checkpointlocations model to predict checkpoint locations using the training set;calculating a checkpoint locations prediction error; and optimizing thecheckpoint locations model using the calculated checkpoint locationsprediction error.
 42. The computer-implemented method of claim 41,wherein the one or more datasets further comprise one or more of:clinical procedure related dataset, patient related dataset andadministrative related dataset.
 43. The computer-implemented method ofclaim 41, wherein the automated medical device related dataset comprisesparameters selected from: entry point, insertion angles, targetposition, target position updates, target contour and/or bounding boxand/or location, “no-fly” zones masks and/or bounding boxes and/orlocations, organs segmentation masks and/or bounding boxes and/orlocations, tissues segmentation mask and/or bounding boxes and/orlocations, blood vessels mask and/or bounding boxes and/or locations,planned trajectory, trajectory updates, real-time positions of themedical instrument, number of checkpoints along the planned and/orupdated trajectory, checkpoint locations, checkpoint locations updates,checkpoint errors, position of the automated medical device, steeringsteps timing, procedure duration, steering phase duration, procedureaccuracy, target error, medical images, medical imaging parameters perscan, radiation dose per scan, total radiation dose in steering phase,total radiation dose procedure, sensor(s) measurements, errors indicatedduring the steering procedure, software logs, motion control traces,automated medical device registration logs, medical instrument detectionlogs, homing and BIT results, or any combination thereof.
 44. Thecomputer-implemented method of claim 42, wherein the clinical procedurerelated dataset comprises parameters selected from: medical proceduretype, target organ, target size, target type, type of medicalinstrument, dimensions of the medical instrument, clinical complicationsbefore, during and/or after the procedure, complications detection time,adverse events before, during and/or after the procedure, respirationsignals of the patient, or any combination thereof; wherein the patientrelated dataset comprises parameters selected from: age, gender, race,medical condition, medical history, vital signs before, after and/orduring the procedure, body dimensions, pregnancy, smoking habits,demographic data, or any combination thereof; wherein the administrativerelated dataset comprises parameters selected from: institution,physician, staff, system serial number, disposable components used inthe procedure, software version, operating system version, configurationparameters, or any combination thereof.
 45. The computer-implementedmethod of claim 42, wherein one or more of the parameters of the one ormore datasets is configured to be collected automatically.
 46. Thecomputer-implemented method of claim 41, further comprising: executingone or more individual models using at least a portion of the one ormore datasets and a checkpoint locations prediction generated by thecheckpoint locations model; and obtaining one or more predictions fromthe one or more individual models.
 47. The computer-implemented methodof claim 46, further comprising: calculating a loss function using acheckpoint locations prediction error and the one or more predictionsgenerated by the one or more individual models; optimizing thecheckpoint locations model using the loss function; and training the oneor more individual models.
 48. The computer-implemented method of anyone of claim 46, wherein the one or more individual models comprise oneor more of: a model for predicting an accuracy of an image-guidedinsertion procedure, a model for predicting a radiation dose emittedduring an image-guided insertion procedure, or part thereof, a model forpredicting a duration of an image-guided insertion procedure, or partthereof, and a model for predicting a risk of an image-guided insertionprocedure.
 49. The computer-implemented method of claim 47, whereincalculating the loss function comprises minimizing one or more of thecheckpoint locations prediction error, the predicted radiation dose, thepredicted duration and the predicted risk, and maximizing the predictedaccuracy of the image-guided insertion procedure.
 50. Thecomputer-implemented method of claim 47, further comprising adjustingone or more coefficients of one or more terms used in the calculation ofthe loss function, the one or more terms being associated with at leastone of the checkpoint locations prediction error and the one or morepredictions generated by the one or more individual models; wherein theadjusting of the one or more coefficients is executed during training ofthe checkpoint locations model; wherein the adjusting of the one or morecoefficients is executed during execution of the checkpoint locationsmodel.
 51. The computer-implemented method of claim 41, wherein theautomated medical device is configured to steer the medical instrumenttoward the target such that the medical instrument traverses anon-linear trajectory within the body of the patient and/or, wherein theautomated medical device is configured to allow real-time updating of atrajectory of the medical instrument.
 52. The computer-implementedmethod of claim 41, further comprising: collecting one or more newdatasets, at least one of the one or more new datasets being related toan automated medical device configured to steer a medical instrumenttoward a target in a body of a patient and/or to operation thereof andcomprising one or more images of a region of interest and a plannedtrajectory for the medical instrument from an entry point to the target;detecting one or more tissue boundaries in the one or more images;executing the checkpoint locations model using at least a portion of theone or more new datasets; obtaining an output of the checkpointlocations model; setting one or more checkpoints along the plannedtrajectory based on the output of the checkpoint locations model, andestimating a scan volume and a radiation dose per checkpoint.
 53. Thecomputer-implemented method of claim 52, further comprising the step ofobtaining from a user at least one of a confirmation to the set one ormore checkpoints and an adjustment thereto.
 54. The computer-implementedmethod of claim 52, further comprising the step of defining one or moresections along the planned trajectory in which no checkpoints are to bepositioned, so as to allow the medical instrument to be continuouslyadvanced along the one or more sections.
 55. The computer-implementedmethod of claim 52, wherein if at least one of a position of the targetand the planned trajectory are updated upon reaching a checkpoint, themethod further comprises the steps of re-executing the checkpointlocations model and obtaining an updated output of the checkpointlocations model; adjusting the locations of one or more subsequentcheckpoints based on the updated output of the checkpoint locationsmodel; and; obtaining from a user at least one of a confirmation to theadjusted locations of the one or more subsequent checkpoints and anadjustment thereto.
 56. A system for utilizing a checkpoint locationsmodel for optimizing locations of checkpoints along a trajectory in animage-guided procedure for inserting a medical instrument to a target ina body of a patient, the system comprising: an inference modulecomprising: a memory configured to store the one or more datasets; andone or more processors configured to: collect one or more datasets, atleast one of the one or more datasets being related to an automatedmedical device configured to steer a medical instrument toward a targetin the body of a patient and/or to operation thereof; create a trainingset comprising at least a portion of the one or more datasets and one ormore target parameters relating to checkpoint locations along atrajectory in one or more previous image-guided procedures for insertinga medical instrument toward a target in a body of a patient; train thecheckpoint locations model to predict checkpoint locations using thetraining set; calculate a checkpoint locations prediction error, andoptimize the checkpoint locations model using the calculated checkpointlocations prediction error.
 57. The system of claim 56, wherein one ormore processors are further configured to real-time update a trajectoryof the medical instrument.
 58. The system of claim 57, wherein if atleast one of a position of the target and the planned trajectory areupdated upon reaching a checkpoint, the one or more processors arefurther configured to re-execute the checkpoint locations model andobtaining an updated output of the checkpoint locations model; adjustthe locations of one or more subsequent checkpoints based on the updatedoutput of the checkpoint locations model; obtain from a user at leastone of a confirmation to the adjusted locations of the one or moresubsequent checkpoints and an adjustment thereto.
 59. The system ofclaim 56, wherein the inference module is located on a remote server, an“on premise” server or a computer associated with the automated medicaldevice.
 60. The system of claim 59, wherein the remote server is a cloudserver.